Comparative evaluation of large-language models and purpose-built software for medical record de-identification

doi:10.21203/rs.3.rs-4870585/v1

Download PDF

Article

Comparative evaluation of large-language models and purpose-built software for medical record de-identification

https://doi.org/10.21203/rs.3.rs-4870585/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background:

Robust de-identification is necessary to preserve patient confidentiality and maintain public acceptability for electronic health record (EHR) research. Manual redaction of personally identifiable information (PII) is time-consuming and expensive, limiting the scale of data-sharing. Automated de-identification could alleviate this burden, but the best strategy is not clear. Advances in natural language processing (NLP) and the emergence of foundational large language models (LLMs) show promise in performing clinical NLP tasks with no, or limited training.

Methods:

We evaluated two task-specific (Microsoft Azure de-identification service, AnonCAT) and five general LLMs (Gemma-7b-IT, Llama-3-8B-Instruct, Phi-3-mini-128k-instruct, GPT3.5-turbo-base, GPT-4-0125) in de-identifying 3650 medical records from a UK hospital group, split into general and specialised datasets. Records were dual-annotated by clinicians for PII. Inter-annotator reliability was used to benchmark performance. The primary outcome was F1, precision (positive predictive value) and recall (sensitivity) for each comparator in classifying words as PII vs. non-PII. The secondary outcomes were performance per-PII-subtype, per-dataset, and the presence of LLM hallucinations. We report outcomes at zero- and few-shot learning for LLMs, and with/without fine-tuning for AnonCAT.

Results:

17496/479760 (3.65%) words were PII. Inter-annotator F1 for word-level PII/non-PII was 0.977 (95%CI 0.957-0.991), precision 0.967 (0.923-0.993), and recall 0.986 (0.971-0.997). The best performing redaction tool was the Microsoft Azure de-identification service: F1 0.933 (0.928-0.938), precision 0.916 (0.930-0.922), recall 0.950 (0.942-0.957). The next-best were fine-tuned-AnonCAT: F1 0.873 (0.864-0.882), precision 0.981 (0.977-0.985), recall 0.787 (0.773-0.800), and GPT-4-0125 (ten-shots): F1 0.898 (0.876-0.915), precision 0.924 (0.914-0.933), recall 0.874 (0.834-0.905). There was hallucinatory output in Phi-3-mini-128k-instruct and Llama-3-8B-Instruct at zero-, one-, and five-shots, and universally for Gemma-7b-IT. Names/dates were consistently redacted by all comparators; there was variable performance for other categories. Fine-tuned-AnonCAT demonstrated the least performance shift across datasets.

Conclusion:

Automated EHR de-identification could facilitate large-scale, domain-agnostic record sharing for medical research, alongside other safeguards to prevent patient reidentification.

In the last decade, there has been increasing digitization of medical data¹. Routinely collected healthcare data from millions of patients are stored as electronic health records (EHR), facilitating efficient data sharing and communication between health providers^1,2. The information stored within EHR is diverse, including structured data such as laboratory results, and semi-structured or unstructured information, such as diagnostic reports. The resulting data-store is a valuable resource for research, audit, education and quality improvement³. Recently, there has been increasing interest in utilizing EHR to develop and validate deep learning models to improve healthcare outcomes⁴.

Sharing data for secondary use necessitates robust de-identification of records to preserve patient confidentiality. Identifiers, such as patient names, must be removed from records to minimize the risk of re-identification, comply with data protection legislation, and establish public trust and acceptability^5,6. In Europe, General Data Protection Regulation defines personal data as any information that relates to an identified or identifiable individual. In the US, identified healthcare records are personally identifiable information (PII) when they contain any of 18 identifiers defined in the Health Insurance Portability and Accountability Act (HIPAA).

Effective redaction of PII from semi- or unstructured medical records can be challenging. Manual de-identification, in which humans review and redact personal information, is time-consuming and costly, limiting the scale of data that can safely be made available⁷. Automated de-identification offers considerable benefits to hospitals, researchers, and patients. However, some categories of PII, such as names and locations are poorly recognized using regular expressions or rule-based algorithms, requiring curation of extensive local dictionaries to ensure adequate redaction^8–10. Rule-based models may be prone to over-redaction, compromising the utility of de-identified data^8,9. Strategies relying on local dictionaries are vulnerable to spelling errors and require re-definition if applied to new datasets¹⁰. There is heterogeneity in clinical language across medical specialties, necessitating domain adaptation to maintain performance.

Domain-agnostic, automated de-identification that requires none, or minimal training could facilitate safe and cost-effective data sharing at scale. Recently, transformer-based natural language processing (NLP) models have shown promise in de-identification without the requirement for domain-specific adaptation^9,11,12. Existing research shows that LLMs may be able to perform clinical/biomedical NLP tasks with no domain-specific training (zero-shot learning), or with a few examples of input and desired output (few-shot learning)^9,13,14. A critical challenge of LLMs is the phenomenon of ’hallucination’, in which LLMs generate erroneous output; e.g., offering opinions on the reference text^15–17.

In this study, we evaluated and compared two existing proprietary de-identification software tools, and five LLMs in the task of text de-identification, using a dataset of 3650 mixed clinical records from a group of National Health Service (NHS) hospitals in the United Kingdom.

Dataset

We used routinely collected data from Oxford University Hospitals NHS Foundation Trust (OUHNFT), a large teaching trust comprised of four hospitals providing care to around 1% of the UK population as well as specialist referral services¹⁸. Data were obtained from two sources between 01/2020-01/2022 inclusive. We randomly sampled 1000 radiology and 1000 histopathology reports from the Infections in Oxfordshire Research Database (IORD). IORD has approvals from the National Research Ethics Service South Central – Oxford C Research Ethics Committee (19/SC/0403), the Health Research Authority and the Confidentiality Advisory Group (19/CAG/0144). Structured identifiers were removed from records prior to analysis, but free-text was provided without further redaction to researchers with NHS contracts for analysis within NHS infrastructure. We randomly sampled 550 radiograph, 550 computed tomography (CT), and 550 magnetic resonance (MR) request and report entries from a specialised musculoskeletal database, with Camden and Kings Cross Research Ethics Committee approval (22/LO/0049).

PII definition and annotation rules

We developed our labelling ontology based on HIPAA PII categories¹⁹, including three further categories we considered additional potential identifiers: hospital/unit names, private healthcare organisation names, and individual professional details, where included as part of a healthcare professionals’ name, or where related to patients (Table 1).

Table 1

Categories of PII.
	PII category	PII subcategory
1	Names	Patient or relative
1	Names	Healthcare worker
2	Address	House or street number
		Street name
		City
		County
		Country
		Postcode
3	Dates	Dates
3	Dates	Age over 89 years
4	Phone numbers
5	Fax numbers
6	Electronic email addresses
7	Social security numbers
8	Medical record numbers	Medical record number
		NHS number
		Specimen identifier
9	Health plan beneficiary numbers
10	Account numbers
11	Certificate/license numbers	GMC number
		NMC number
		Any other license
12	Vehicle identifiers and serial numbers
13	Device identifiers and serial numbers
14	Web Universal Resource Locators (URLs)
15	Internet Protocol (IP) address numbers
16	Any other unique identifying number, characteristic, or code	Hospital/unit
		External healthcare organisation
		Professional details

We excluded two inapplicable HIPAA categories (biometric identifiers and full-face photographic images).

All examples were annotated by two clinically qualified labellers (two of RK, CO, and AH), using the open-source software doccano, hosted on an OUHNFT server²⁰. Annotations were applied on a word-level and included PII category, and the start and end position of each annotation. After completion of labelling, any disagreements were recorded, and then resolved with discussion.

Deidentification models

We evaluated models that require no, or minimal training for de-identification including two purpose-built de-identification software tools (Microsoft Azure de-identification service, AnonCAT^21,22) and five LLMs (Gemma-7b-IT, Llama-3-8B-Instruct, Phi-3-mini-128k-instruct, GPT3.5-turbo-base and GPT4-0125²¹⁻²⁵; details in Supplement). All 3650 examples were used for evaluating performance. For each LLM, we compared performance with zero, one-, five- and ten-shot learning, using examples sourced from outside the dataset (Supplement).

We evaluated AnonCAT without fine-tuning, and with fine-tuning. To fine-tune AnonCAT, we used a randomly selected sample of 365 (10%) of the annotated dataset, stratified per dataset (Supplement). We performed our evaluation using the remaining 3285 documents. All models were accessed using OUHNFT’s secure Azure tenancy between 04/2024-06/2024.

Prompt structure

Optimising LLM prompts improves downstream task performance²⁶. We provided each LLM with a structured prompt. First, we specify the task (‘Anonymise the following text’). Second, we specify rules for task completion (’Replace all identifiers with their classification’). Third, we provide the labelling ontology, and a brief, synthetic example of desired output (‘If you find a doctor’s name, such as ‘Dr John Doe’, replace this with [profession] [ doctor] [doctor]’). Finally, we describe undesirable output (’Do not output any additional text’).

Statistical analysis and evaluation metrics

We estimate word-level inter-annotator reliability of clinicians using precision (positive predictive value of a PII label), recall (sensitivity, the proportion of all PII detected) and pairwise F1 scores (harmonic mean of precision and recall), using two prediction classes (PII/non-PII)²⁷. One clinician (RK) was chosen as the gold standard, and precision/recall were calculated using this assumption. Precision and recall are interchanged depending on which annotator is designated as the gold standard. The calculation of pairwise F1 score is independent of the designation of annotators as the gold standard or experimental source²⁷.

Software and LLM model performance was compared to a composite reference standard combining two clinician annotations; we report precision, recall and F1 score for two prediction classes (PII/non-PII). We considered that some, but not all, uses of secondary data would necessitate the redaction of professional titles where this was related to healthcare professionals. This was reflected by the two proprietary models, Microsoft Azure de-identification service and AnonCAT, which did, and did not, redact professional titles respectively. We performed additional analyses to report AnonCAT performance with, and without redaction of professional titles.

To describe per-category recall, we used clinician-assigned PII categories, considering PII to be correctly identified if it was redacted, even where models classified it differently. This reflects different proprietary models have different labelling ontologies, and that redaction rather than PII classification is most important in real-world settings. We calculate recall for all categories which comprise > 1% of all PII in the dataset.

We defined LLM hallucinations as additional text that was not present in the reference text. To evaluate hallucinations, we calculated BLEU scores and Levenshtein distance, commonly used metrics for measuring string similarity^28,29. The Levenshtein distance is defined as the minimum number of character-edits required to transform one string to another; here, from redacted to reference records²⁸. BLEU scores are defined by n-gram overlap between two strings; here, we calculated the cumulative 4-gram BLEU score²⁹. Low BLEU scores and high Levenshtein distances are potentially indicative of LLM hallucination. We qualitatively examinated 50 randomly selected samples of each LLM output, at each iteration of zero- and few-shot learning, to confirm/disconfirm these findings. We calculated 95% confidence intervals (CI) using bootstrapping with 10,000 samples. We coded all analyses using Python (v3.8.10).

Dataset description

We analysed 3605 records consisting of 479760 words, of which 17496 (3.65%) were PII. Record length and PII prevalence differed across datasets (Table S1). The most frequent forms of PII were names (6901/17496, 39.4%), of which the majority were healthcare professional names (6870/6901, 99.6%) (Table S2). In common with previous research, only a minority of names were patient names (31, 0.4%)³⁰.

The next most frequent PII categories were ‘other unique identifiers’ (4758, 27.2%; comprising of professional details, names of external healthcare organisations, and names of hospitals or healthcare units), dates (3641, 20.8%), medical record numbers (1408, 8.0%), and telephone numbers (334, 1.9%). All other PII categories had a prevalence of < 1%. There were no occurrences of fax numbers, health plan beneficiary numbers, vehicle/device identifiers, or IP addresses.

Inter-annotator results

There was excellent agreement between clinician annotators. Pairwise-F1 for classification of PII/non-PII was 0.977 (0.957–0.991), precision 0.967 (0.932–0.993), and recall 0.986 (0.971–0.997) (Table S2). All discrepancies between annotators were due to cases in which one annotator did not notice a PII-word. Once identified, there were no disagreements about whether a word should be classified as PII. The BLEU score between the original, unredacted records, and manually redacted records was 0.931 (0.923–0.934), reflecting the prevalence of PII in the dataset (Table 2). The Levenshtein distance was 67.0(63.3–70.8).

Table 2

**String similarity between original and redacted text.** Per model BLEU scores and Levenshtein distances are shown.
Model type	Model name	Number of shots	BLEU score (95% CI)	Levenshtein distance (95% CI)
Inter-annotator		N/A	0.931 (0.923–0.934)	67.0 (63.3–70.8)
Proprietary de-identification software	Microsoft Azure de-identification service	N/A	0.929 (0.927–0.932)	35.7 (34.1-37.49)
	AnonCAT	No fine-tuning	0.948 (0.946–0.950)	30.2 (28.9–31.6)
	AnonCAT	Fine-tuned	0.948 (0.946–0.950)	29.3 (28.0-30.6)
Large language models	Gemma-7b-IT	0	0.071 (0.068–0.075)	749.3 (726.1-773.9)
		1	0.032 (0.030–0.035)	860.4 (838.1-883.8)
		5	0.025 (0.023–0.026)	758.2 (735.6-781.6)
		10	0.031 (0.029–0.033)	741.8 (717.1-765.33)
	Llama-3-8B-Instruct	0	0.259 (0.250–0.269)	465.0 (448.7-482.2)
		1	0.500 (0.491–0.510)	145.2 (139.4–151.0)
		5	0.769 (0.760–0.778)	71.3 (67.2–75.1)
		10	0.693 (0.682–0.703)	106.0 (99.3-113.5)
	Phi-3-mini-128k-instruct	0	0.396 (0.383–0.407)	836.3 (790.6-882.7)
		1	0.515 (0.498–0.533)	603.8 (566.2–640.0)
		5	0.482 (0.468–0.497)	384.8 (369.0-400.3)
		10	0.663 (0.649–0.676)	281.4 (262.4-302.1)
	GPT3.5-turbo-base	0	0.838 (0.832–0.843)	87.6 (83.9–92.1)
		1	0.878 (0.873–0.883)	73.0 (69.4–77.2)
		5	0.926 (0.921–0.930)	51.6 (47.9–56.2)
		10	0.932 (0.928–0.936)	47.2 (41.1–52.1)
	GPT-4-0125	0	0.920 (0.915–0.924)	50.5 (46.5–55.3)
		1	0.922 (0.917–0.927)	51.4 (47.3–56.1)
		5	0.925 (0.921–0.929)	54.3 (49.7–59.2)
		10	0.920 (0.915–0.924)	52.6 (48.4–57.3)

Model results

PII vs. non-PII

There was substantial variation in performance between comparators (Fig. 1, Table S3). The Microsoft Azure de-identification service had the highest F1 score 0.933 (95%CI 0.928–0.938), precision of 0.916 (0.930 − 0.922), and recall of 0.950 (0.942–0.957), approaching clinician performance. The fine-tuned AnonCAT (FT-AnonCAT) model had an F1 score 0.873 (0.864–0.882), precision 0.981 (0.977–0.985) and recall 0.787 (0.773-0.800), when not including redaction of healthcare professional titles, such as ‘Dr’. With the requirement to redact professional titles, the performance of the FT-AnonCAT model was lower, with F1 score 0.800 (0.843–0.858), precision 0.981 (0.977–0.985) and recall 0.676 (0.665–0.686).

The best performing LLM was GPT-4-0125 with ten-shot learning, with F1 score 0.898 (0.876–0.915), precision 0.924 (0.914–0.933), and recall 0.874 (0.834–0.905)(Figure S1). This was followed by GPT-3.5-turbo-base with ten-shot learning, with F1 score 0.831 (0.807–0.851), precision 0.856 (0.812–0.892), and recall 0.807 (0.788–0.825).

There was improvement in GPT3.5-turbo-base with few-shot learning: the F1 score rose from 0.530 (0.514–0.547) at zero shots to 0.831 (0.807–0.851) with 10 shots, driven by improved precision; recall remained similar through all iterations of zero- and few-shot learning. On qualitative examination with none, or fewer in-context examples, the LLM over-redacted records, including clinically relevant information such as diagnosis, or details of pathology.

The performance of other models was more modest. The next best was Phi-3-mini-128k-instruct, also improving with few-shot learning, with F1 score 0.146 (0.140–0.153) at zero-shots, improving to 0.448 (0.430–0.467) with ten-shots. However, there was significant imbalance between precision and recall. At ten-shots, precision was 0.297 (0.282–0.314) and recall 0.904 (0.892–0.915). This was consistent with our findings on qualitative examination, showing over-redacted records.

Llama-3-8B-Instruct showed the same pattern of over-redaction, showing best performance at five-shots, with an F1 0.198 (0.181–0.216), precision 0.077 (0.069–0.085) and recall 0.990 (0.983–0.995). We did not observe any change in performance from zero- to few-shot learning with Gemma-7b-IT. The F1 at zero-shots was 0.089 (0.086–0.092), and at ten-shots 0.041 (0.037–0.044). At 10-shots, precision was 0.021 (0.019–0.023) and recall was 0.905 (0.885–0.923). Qualitative examination Gemma-7b-IT output showed hallucinatory content was universally present.

Evaluation of text similarity and LLM hallucinations

BLEU scores were high, reflecting close text similarity post-redaction to the original text, for both proprietary models, the Microsoft Azure de-identification service and FT-AnonCAT, 0.929 (0.927–0.932) and 0.948 (0.946–0.950), respectively. This was similar to the BLEU score recorded between clinician redaction and reference text (Table 2). The Levenshtein distances for the Microsoft Azure deidentification service and FT- AnonCAT were 35.74 (34.13–37.5) and 29.3 (28.0-30.6), both lower than the distance reported for clinician redaction. Qualitative examination of the output of both proprietary models showed no evidence of hallucination.

The best performing LLMs, GPT-4-0125 and GPT3.5-turbo-base, had consistently similar BLEU scores and Levenshtein distances to values recorded for clinician redaction, and showed no evidence of hallucination on qualitative examination.

Both Phi-3-mini-128k-instruct and Llama-3-8B-Instruct showed improved BLEU scores and Levenshtein distances across zero- and few-shot learning. On qualitative examination, we confirmed that both Phi-3-mini-128k-instruct and Llama-3-8B-Instruct showed evidence of hallucinatory behaviour at zero-, one- and five-shot learning. This included explanations of the task or output (e.g., ‘This text does not contain explicit identifiers, therefore the text remains unchanged’) alongside nonsensical string (e.g., long spans of punctuation). We did not observe any hallucinations at ten-shots.

We report consistently low BLEU scores and high Levenshtein distances across zero- and few-shot learning for Gemma-7b-IT. The output was grossly hallucinatory, including hallucinated medical history (‘Historical factors include prior trauma-related injury sustained one month back’), translations into other languages, and treatment recommendations (‘She’ll have an appointment to see her Dr tomorrow so we can discuss it then’). We therefore did not include a further evaluation of recall for individual PII categories for Gemma-7b-IT.

PII redaction per category

Names and dates were redacted consistently by all models (Table 4). However, medical record numbers, phone numbers, and the other unique identifiers had variable redaction across models. The Microsoft Azure de-identification service, GPT-4-0125, Llama-3-8B-Instruct and Phi-3-mini-128k-instruct had consistently high recall across PII categories. For these models, the lowest recall was for ‘other unique identifiers’. GPT3.5-turbo-base and FT- AnonCAT had more variation in recall per PII category. Both GPT3.5-turbo and FT- AnonCAT had lower recall for ‘other unique identifiers’, at 0.676 (0.648–0.705) and 0.575 (0.542–0.610) respectively.

Table 4

**Recall per PII category**. Results for AnonCAT are shown for fine-tuning with 365 examples; results for LLMs are shown using 10 shot learning.
Model type	Model name	Names	Other unique identifiers	Dates	Medical record numbers	Phone numbers
Proprietary de-identification software	Microsoft Azure de-identification service	0.985 (0.981–0.989)	0.890 (0.870–0.909)	0.984 (0.979–0.989)	0.948 (0.933–0.963)	0.972 (0.954–0.988)
Proprietary de-identification software	FT-AnonCAT	0.883 (0.870–0.896)	0.575 (0.542–0.610)	0.917 (0.901–0.932)	0.884 (0.853–0.914)	0.966 (0.945–0.984)
Large language models	Llama-3-8B-Instruct	1.00 (1.00–1.00)	0.932 (0.891–0.966)	0.980 (0.963–0.993)	0.989 (0.971-1.000)	1.000 (1.000–1.000)
	Phi-3-mini-128k-instruct	0.978 (0.969–0.985)	0.814 (0.790–0.837)	0.940 (0.926–0.953)	0.973 (0.961–0.983)	0.983 (0.963–0.997)
	GPT-3.5-turbo-base	0.911 (0.893–0.928)	0.676 (0.648–0.705)	0.882 (0.860–0.902)	0.866 (0.840–0.890)	0.935 (0.900-0.967)
	GPT4-0125	0.988 (0.983–0.992)	0.841 (0.818–0.862)	0.970 (0.960–0.979)	0.966 (0.953–0.977)	0.994 (0.986-1.000)

PII redaction per dataset

Of the best performing models, AnonCAT had the least performance shift per dataset, followed by the Microsoft Azure de-identification service (Fig. 2, Table S4). GPT-4-0125, the best performing LLM, had a wide range of performance across datasets, with highest performance in the general histopathology dataset, with an F1 0.949 (0.938–0.958), and lowest in the musculoskeletal radiograph dataset, with an F1 0.672 (0.580–0.744). Likewise, GPT-3.5-turbo-base, Llama-3-8B-Instruct and Phi-3-mini-128k-Instruct had more performance shift across datasets than both proprietary models.

In this study, we evaluated the performance of two off-the-shelf software tools and five LLMs in de-identification of unstructured and semi-structured clinical text, using a dataset of 3650 individual records. Our results suggest that automated de-identification, using either purpose-built software or foundational LLMs, have performance approaching that of human clinicians. However, both human and model-based redaction while good, is not perfect. A single clinician identified 98.6% of all PII, with 96.7% of words redacted representing true PII. The best performing tool overall, Microsoft’s deidentification service identified 95% of PII, including 98.5% of all names, and 91.6% of words redacted represented true PII.

The best-performing LLMs in this study de-identified clinical notes with high performance using zero- or few-shot learning, suggesting that it is feasible to avoid or reduce the burden of curating large, manually labelled datasets for model training. However, in all models, we report performance shift across domains. In general, performance was highest in semi-structured histopathology reports, compared to unstructured radiology reports. Both proprietary models demonstrated less performance shift across datasets than LLMs.

There was variable performance across LLMs. Both Phi-3-mini-128k-instruct and Llama-3-8B-IT over-redacted records: this is reflected by consistently high recall in all domains and across all datasets, but low precision and F1 scores. Qualitative examination of records redacted by both LLMs revealed that clinically relevant information was redacted incorrectly; for example, diagnoses or symptoms, reducing the downstream utility of the redacted records. Here, Gemma-7b-IT was particularly prone to hallucination. Qualitative examination showed a mixture of hallucinations, ranging from explaining the task, to offering clinical recommendations. This output is not useful for secondary research and may be actively harmful.

Our study has several limitations. The data from this study derived from a single hospital group in the United Kingdom, and software and LLM performance is likely to differ by location. Our dataset comprises only English-language clinical notes, and we cannot comment on the performance of automated de-identification in other languages. All LLMs in this study were pretrained on an English-dominant corpus, and may transfer differentially to other languages, particularly low-resource languages^31–33.

We were unable to evaluate redaction of uncommon, but critical, categories of PII; e.g., email addresses. Furthermore, we were unable to comment on sub-categories of PII, including specific performance for patient names, due to the low prevalence in our dataset. We qualitatively examined 50 examples of output for each model, per-shot, for hallucination, but cannot conclusively report on the prevalence of hallucination for the entire dataset. Finally, we report on seven different comparators, but since conducting the analyses, other software or LLMs may have been developed with superior performance.

A fundamental issue is what constitutes acceptable performance. Although attractive to strive for perfect redaction, the human performance and inter-annotator variability we describe shows this is unlikely to be possible, without significant extra resource. One approach is to see what precedents exist. The MIMIC datasets are amongst the largest publicly available datasets to date. These are redacted by a combination of removing all entries from a database of identifiers, e.g. names, as well as by other string-pattern matching. The recall of this approach on a test corpus was 0.943¹⁰. Similar performance was achieved by the Microsoft Azure de-identification service and GPT-4-0125, with high precision, in our analysis. Future work could include hybrid approaches based on databases of identifiers, on a population, institution or personal level, to enhance redaction of these alongside generic model-based approaches.

If redaction is imperfect, other protections are required to protect privacy and confidentiality. These include requiring researchers to not attempt re-identification of individuals, to only access sufficient data to answer research questions, and retaining data within secure data environments, limiting access to approved researchers.

Within the scope of these protections, automating de-identification of clinical notes could enable large-scale sharing of clinical data and its use in training large-scale models, with less time and expense than manual de-identification. Use of domain-agnostic, off-the-shelf models, or LLMs requiring only a few in-context examples, would further reduce the cost of implementing automated de-identification. Our findings illustrate the potential for automated de-identification systems, but we urge caution in deployment, particularly in implementing systems based on LLMs. Careful examination of model output, across clinical and PII domains, with attention to hallucinatory output and loss of critical clinical information through over-redaction is crucial to ensure the reliability of automated de-identification in real-world practice.

Author contributions

RK and AS conceptualized the study, curated data, gained ethical approval and study funding, and performed data analyses and interpretation. RK wrote the manuscript. AS was a major contributor in writing the manuscript. CO and AH curated data and contributed to writing the manuscript. DC and GC provided methodological oversight of the study and contributed to data interpretation. DF and DE contributed to conceptualization of the study, data acquisition and curation, provided methodological oversight and contributed to writing the manuscript. All authors read and approved of the final manuscript.

Acknowledgements

Dr Rachel Kuo is supported by a National Institute for Health Research (NIHR) Doctoral Research Fellowship (NIHR302562). Dr Andrew Soltan is supported by the Microsoft Accelerating Foundation Models grant. Professor David Clifton is supported by the Pandemic Sciences Institute at the University of Oxford; the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC); an NIHR Research Professorship; a Royal Academy of Engineering Research Chair; the Wellcome Trust funded VITAL project (grant 204904/Z/16/Z); the EPSRC (grant EP/W031744/1); and the InnoHK Hong Kong Centre for Cerebro-cardiovascular Engineering (COCHE). Professor Gary Collins is supported by Cancer Research UK (programme grant: C49297/A27294), and by EPSRC grant for ‘Artificial intelligence innovation to accelerate health research’ (number: EP/Y018516/1, and is a National Institute for Health and Care Research (NIHR) Senior Investigator. The views expressed in this article are those of the author and not necessarily those of the NIHR, or the Department of Health and Social Care. Professor Dominic Furniss is supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC), and UKRI (MR/Y030419/1). Professor David Eyre is a Robertson Foundation Fellow. This study was also supported by the NIHR Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Oxford University in partnership with the UK Health Security Agency (UKHSA) and the NIHR Biomedical Research Centre, Oxford. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research, the Department of Health and Social Care or the UKHSA.

We would like to thank Kimia Mavon (Microsoft Research UK) for provision of the Microsoft Azure de-identification service, and Richard Dobson and Xi Bai (CogStack) for provision of AnonCAT, free-of-charge for the purposes of this research. Neither party, and none of the funders had input into study design, data collection, data annotation, data analysis, interpretation of results or writing of the manuscript.

Henry J, Pylypchuk Y, Searcy T, Patel V. Adoption of electronic health record systems among US non-federal acute care hospitals: 2008–2015. ONC data brief 2016;35(35):2008-2015.
Slawomirski L, Lindner L, de Bienassis K, Haywood P, Hashiguchi TCO, Steentjes M, et al. Progress on implementing and using electronic health record systems: Developments in OECD countries as of 2021. 2023.
Shah SM, Khan RA. Secondary use of electronic health record: Opportunities and challenges. 2020;8:136947-136965.
Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. 2018;25(10):1419-1428.
Kalkman S, van Delden J, Banerjee A, Tyl B, Mostert M, van Thiel G. Patients’ and public views and attitudes towards the sharing of health data for research: a narrative review of the empirical evidence. J.Med.Ethics 2022;48(1):3-13.
Kuo RYL, Freethy A, Smith J, Hill R, Joanna C, Jerome D, et al. Stakeholder perspectives towards diagnostic artificial intelligence: a co-produced qualitative evidence synthesis. 2024;71.
Computer-assisted de-identification of free text in the MIMIC II database. Computers in Cardiology, 2004: IEEE; 2004.
Steinkamp JM, Pomeranz T, Adleberg J, Kahn Jr CE, Cook TS. Evaluation of automated public de-identification tools on a corpus of radiology reports. 2020;2(6):e190137.
Kovačević A, Bašaragin B, Milošević N, Nenadić G. De-identification of clinical free text using natural language processing: A systematic review of current approaches. Artif.Intell.Med. 2024:102845.
Neamatullah I, Douglass MM, Lehman LH, Reisner A, Villarroel M, Long WJ, et al. Automated de-identification of free-text medical records. 2008;8:1-17.
Instruction-guided deidentification with synthetic test cases for Norwegian clinical text. Northern Lights Deep Learning Conference: PMLR; 2024.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. 2017;30.
Labrak Y, Rouvier M, Dufour R. A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks. 2023.
Liu Z, Huang Y, Yu X, Zhang L, Wu Z, Cao C, et al. Deid-gpt: Zero-shot medical text de-identification by gpt-4. 2023.
Xu Z, Jain S, Kankanhalli M. Hallucination is inevitable: An innate limitation of large language models. 2024.
Huang L, Yu W, Ma W, Zhong W, Feng Z, Wang H, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. 2023.
Rawte V, Sheth A, Das A. A survey of hallucination in large foundation models. 2023.
OUHNFT. Oxford University Hospitals NHS Foundation Trust. Available at: https://www.ouh.nhs.uk/about/. Accessed June, 2024.
Act A. Health insurance portability and accountability act of 1996. 1996;104:191.
Hironsan. doccano: an open-source text annotation tool. . Accessed June, 2024.
Mavon K. Microsoft De-Identification Service. . Accessed June, 2024.
Validating transformers for redaction of text from electronic health records in real-world healthcare. 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI): IEEE; 2023.
Team G, Mesnard T, Hardin C, Dadashi R, Bhupatiraju S, Pathak S, et al. Gemma: Open models based on gemini research and technology. 2024.
Abdin M, Jacobs SA, Awan AA, Aneja J, Awadallah A, Awadalla H, et al. Phi-3 technical report: A highly capable language model locally on your phone. 2024.
Liu Q, Hyland S, Bannur S, Bouzid K, Castro DC, Wetscherek MT, et al. Exploring the Boundaries of GPT-4 in Radiology. 2023.
Wang J, Shi E, Yu S, Wu Z, Ma C, Dai H, et al. Prompt engineering for healthcare: Methodologies and applications. 2023.
Hripcsak G, Rothschild AS. Agreement, the f-measure, and reliability in information retrieval. 2005;12(3):296-298.
Yujian L, Bo L. A normalized Levenshtein distance metric. IEEE Trans.Pattern Anal.Mach.Intell. 2007;29(6):1091-1095.
Bleu: a method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics; 2002.
Steinkamp JM, Pomeranz T, Adleberg J, Kahn Jr CE, Cook TS. Evaluation of automated public de-identification tools on a corpus of radiology reports. 2020;2(6):e190137.
Zhao J, Zhang Z, Zhang Q, Gui T, Huang X. Llama beyond english: An empirical study on language capability transfer. 2024.
Li Z, Shi Y, Liu Z, Yang F, Liu N, Du M. Quantifying Multilingual Performance of Large Language Models Across Languages. 2024.
Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, et al. Gpt-4 technical report. 2023.

There is a conflict of interest We would like to thank Kimia Mavon (Microsoft Research UK) for provision of the Microsoft Azure de-identification service, and Richard Dobson and Xi Bai (CogStack) for provision of AnonCAT, free-of-charge for the purposes of this research. Neither party had input into study design, data collection, data annotation, data analysis, interpretation of results or writing of the manuscript.

AS is supported by the Microsoft Accelerating Foundation Models grant (£10000), of which £1000 was used to pay for computing costs for this work. Microsoft Research UK did not have any input into study design, data collection, data annotation, data analysis, interpretation of results or writing of the manuscript.

Supplement.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Comparative evaluation of large-language models and purpose-built software for medical record de-identification

Status:

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussion

Declarations

Author contributions

Acknowledgements

References

Additional Declarations

Supplementary Files

Status:

Version 1