Reliability and validation of an electronic penicillin allergy risk-assessment tool in a pregnant population

doi:10.21203/rs.3.rs-3373805/v1

Download PDF

Research Article

Reliability and validation of an electronic penicillin allergy risk-assessment tool in a pregnant population

https://doi.org/10.21203/rs.3.rs-3373805/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 19 Oct, 2024

Read the published version in Allergy, Asthma & Clinical Immunology →

You are reading this latest preprint version

Background:

Penicillin allergy adversely impacts patient care, yet most cases do not have true allergies. Due to high reported prevalence of penicillin allergy, clinicians require efficient, reliable clinical tools to identify low risk patients who can be safely de-labeled. Our center implemented the FIRSTLINE electronic point-of-care decision support tool to help practitioners risk stratify patients with penicillin allergy. We sought to explore the reliability and validity of this tool in relation to actual patient outcomes. We additionally compared it with two other published stratification tools: JAMA and PENFAST, to assess ability to accurately identify low risk patients appropriate for direct oral challenge.

Methods:

In this single-center, retrospective, observational study, 181 pregnant females with self-reported penicillin allergy between July 2019 to June 2021 at BC Women’s Hospital, Vancouver, Canada were used to assess the reliability and validity of all three tools. Physician-guided history of penicillin use and symptoms were used for scoring. Results and recommendations were compared to actual patient outcomes after clinician decision for direct oral challenge or intradermal tests. We additionally applied two other tools to the patient population and compared performance across the tools.

Results:

176 (97.2%) patients were assessed and deemed not allergic. Compared to the gold standard of clinician judgement, our electronic assessment tool has a specificity 0.93(0.88, 0.97), sensitivity 0.65 (0.41, 0.85), positive predictive value (PPV) 0.54 (0.33, 0.74), negative predictive value (NPV) 0.96 (0.91, 0.98). Other clinical tools were applied to the same population with the following results: 1) PENFAST specificity 0.85 (0.79, 0.90), sensitivity 0.40 (0.19, 0.64), PPV0.25 (0.11, 0.43), and NPV 0.92 (0.86, 0.96) and 2) JAMA toolkit specificity 0.66 (0.58, 0.73), sensitivity 0.85 (0.62, 0.97), PPV 0.24 (0.14, 0.35) and NPV 0.97 (0.92, 0.99).

Conclusion:

We demonstrate that our point-of-care electronic algorithm is reliable at identifying low risk pregnant patients, as compared to the gold standard of allergist assessment. To our knowledge, this is the first study to provide direct comparison between multiple decision support tools using the same population, minimizing participant bias. Providing clinical algorithms to accurately identify patients with low risk penicillin allergy, can enable healthcare professionals to safely risk stratify individuals to direct penicillin oral challenges versus referral to specialists. This increases the generalizability and efficiency of penicillin allergy de-labelling.

Penicillin allergy

penicillin de-labelling

FIRSTLINE

PENFAST

JAMA tool

Unconfirmed beta-lactam antibiotic allergies adversely impact patient care as using alternative antibiotics lead to increased risk of multi-drug resistant bacterial infections, prolonged hospitalization, and greater healthcare utilization¹. Reported international penicillin allergy prevalence is approximately 10%, with most cases not having clinically significant hypersensitivity². A systematic review revealed 96.5% of patients could be de-labelled without serious adverse reactions³. This is further supported by recent evidence whereby low risk penicillin allergy patients can safely receive direct oral challenge⁴. Investigating penicillin allergy can be time and resource intensive, including an allergist referral, intradermal skin testing (IDT), and subsequent direct oral challenge (DOC). Our group previously described safety of DOC in pregnant people with low risk penicillin allergy histories⁵.

Due to the high prevalence of reported penicillin allergy and limited allergist availability, clinicians require an efficient, reliable approach to identify low risk patients who can be safely de-labeled. We have previously published a study showing that various healthcare professionals were able to safely and accurately risk stratify penicillin allergies with an electronic clinical algorithm in pediatric patients⁶. The electronic algorithm has been adapted for clinical use through an electronic decision support tool hosted by an antimicrobial stewardship mobile platform: FIRSTLINE (https://app.firstline.org/en/clients/39-bc-womens-hospital/steps/6158). Other risk stratifying algorithms exist, including: JAMA toolkit, a manual stratification and management tool, and PENFAST, a four-question tool providing the risk of true penicillin allergy (Table 1.)^6–8.

Table 1

Summary description of each penicillin de-labeling tool.
Penicillin De-labelling Tool	Description
PENFAST	Four question clinical decision tool which risk stratifies patients to inform clinicians on rate of positive penicillin allergy test based on its five-point scoring system.
FIRSTLINE	Mobile platform for penicillin risk assessment with point of care answer entry based on patient history, providing immediate risk assessment and management recommendations to de-label low risk patients without need for diagnostic testing or allergist assessment.
JAMA	Risk stratification table based on patient history and symptoms, providing immediate risk assessment and management recommendations including direct oral challenge, skin prick test, and referral to allergist.

For the pregnant population, many are hesitant to proceed with penicillin allergy de-labeling despite previous published literature on the safety of DOC during pregnancy⁹. To our knowledge there has been no specificity nor sensitivity comparison between the various de-labelling tools to inform practitioners on their ability to effectively identify low risk individuals.

In this study, we sought to explore the reliability and validity of our electronic point-of-care decision support tool compared to the gold standard of allergist assessment. We additionally compared the ability of the decision support tool to two other published stratification tools to identify patients at low risk of true penicillin allergy: JAMA, and PENFAST, to assess their ability to accurately identify low risk patients appropriate for direct oral challenge. We hypothesized that our electronic tool would be reliable and have high specificity for identifying low risk patients when compared with other decision support tools.

This is a single-center, retrospective, observational study with 181 pregnant females between July 2019 to June 2021. Patients with reported penicillin allergy were referred to the BC Women’s Hospital Penicillin Allergy clinic in Vancouver, Canada and were included regardless of medical, surgical, or medication history. Self-reported allergy histories required for each tool were routinely collected prior to patients’ first clinic visit. Information was stored on REDCap.

The primary outcome of this study was to assess reliability of the electronic decision support tool compared with gold standard assessment. Allergist assessment and outcome to pursue IDT and DOC were previously recorded. Allergy histories were applied to the electronic decision support tool and the risk outcome was compared with allergist assessment. Risk categories included: allergic, very low risk of allergy, possible allergy, not allergic. The secondary outcome was to compare our electronic decision support tool with other published assessment tools to identify low risk patients appropriate for DOC. Patients were risk stratified by each tool into high, intermediate, and low risk categories.

With our interest specifically to identify patients at lowest risk for penicillin allergy, we opted to categorize all patients in the lowest risk stratification tier of each tool as screen negative this included those scoring PENFAST = 0 and 1, JAMA = low risk, FIRSTLINE = very low risk. All remaining patients were considered test screen positive. All patients deemed very low risk by clinician judgement received DOC. All remaining patients received IDT. If IDT was negative, patients then proceeded to a DOC. The screening tool results were compared to actual patient outcomes after IDT or DOC. All equivocal IDT and delayed skin rash were considered test positive. All equivocal DOC without specifiers were considered test negative. Examples of delayed DOC were those with delayed rash (Type 4 hypersensitivity). There were no specifiers for equivocal reactions collected.

The assessor was blinded to patient information during data analysis. An ethics and consent waiver from the University of British Columbia was obtained for this quality improvement project.

The study included 181 pregnant females between 22–48 years-old, referred by obstetricians, general practitioners, registered midwives, or pharmacists (Supplementary material). Patient reported reaction history was collected (Table 2). Most patients had reactions over ten years ago and cannot recall details of their reaction nor which penicillin drug was used. The most reported adverse reactions were maculopapular rashes, followed by urticaria. Based on clinical judgement, 20 patients received IDT prior to DOC (Table 3). Of these, one patient (0.6%) had a positive IDT. 161 (89.0%) patients received DOC, of this, four patients had delayed rashes and were deemed allergic. In total, five patients were deemed allergic, however, no patient experienced anaphylaxis. Four patients with mild symptoms thought to be unrelated to the DOC were considered not allergic. One patient with equivocal DOC with repeated delayed emesis was thought secondary to pregnancy and therefore also considered not allergic. Overall, 176 (97.2%) patients were deemed not allergic by an allergist.

Table 2

Patient phenotype based on self-reported penicillin history. ^aOther symptoms including: face swelling, teeth discoloration, cold sensation, blurred vision, subjective weakness, feet swelling, canker sore, throat discomfort, lost voice, headache, HSP. ^bOther treatments included: Epinephrine: 0, Antihistamines: 22, salbutabmol: 0, IV fluids: 1, steroids: 2, other: 12 (given another steroid, topical steroids, calamine lotion, unclear)
			Number of patients (%)
Type of penicillin used		Amoxicillin	50 (27.6%)
		Amoxicillin-Clavulanate	2 (1.1%)
		Don’t know	56 (30.3%)
		Other penicillin	73 (40.3%)
Time of reaction		> 10 years ago	145 (80.1%)
		> 5 years ago	16 (8.8%)
		13mo to 5 years ag0	8 (4.4%)
		7 to 12 months ago	1 (0.5%)
		3 to 6 months ago	1 (0.5%)
		Don't know	10 (5.5%)
Number of doses till reaction onset		> 7 days	4 (2.2%)
		4 to 7 days	10 (5.5%)
		1 to 3 days	29 (16.0%)
		1 dose	23 (23%)
		Don’t know	110 (60.8%)
		Reaction already present prior	1 (0.5%)
		Blank	4 (2.2%)
Time of symptom onset after most recent dose		< 1 hour	14 (7.7%)
		1 to 2 hours	4 (2.2%)
		3 to 12 hours	12 (6.6%)
		13 to 24 hours	11 (6.1%)
		> 24 hours	1 (0.5%)
		Don’t know	137 (75.7%)
		Reaction already present prior	1 (0.5%)
		Blank	1 (0.5%)
Patient reported reaction	Skin	Macular/papular rash	89 (49.2%)
		Urticaria	59 (32.6%)
		Angioedema	11 (6.1%)
		Blistering/peeling skin or mucous membrane	1 (0.5%)
		Generalized pustulosis	0 (0%)
		Erythema multiforme	0 (0%)
	Respiratory	Cough	0 (0%)
		Wheeze	3 (1.7%)
		Stridor	0 (0%)
		Breathing difficulties	6 (3.3%)
	Gastrointestinal	Nausea	3 (1.7%)
		Vomiting x1	8 (4.4%)
		Vomiting multiple times	2 (1.1%)
		Abdominal discomfort	0 (0%)
		Diarrhea	1 (0.5%)
	Other	Palpitations	0 (0%)
		Decreased level of consciousness	1 (0.5%)
		Arthritis/arthralgia	2 (1.1%)
		Unexplained fever	1 (0.5%)
		Liver or kidney involvement	0 (0%)
		Other^a	20 (11.0%)
Treatment given		No	65 (35.9%)
		Yes^b	31 (17.1%)
		Don’t know	83 (45.9%)
		Blank	2 (1.1%)

Table 3

Clinician judgement to pursue intradermal skin test versus organ challenge against penicillin allergy result. Result + = penicillin allergic, Result - = penicillin nonallergic
	Result +	Result -	Total
Intradermal test	1 (0.6%)	19 (10.5%)	20
Oral challenge	4 (2.2%)	157 (86.7%)	161
Total	5	176	181

Based on the above results, our electronic assessment tool has a sensitivity 0.65 (95% CI 0.41, 0.85), specificity 0.93 (095% CI.88, 0.97), PPV 0.54 (95% CI 0.33, 0.74), NPV 0.96 (095% CI .91, 0.98) when compared with the gold standard of clinician judgement (Table 4.). When other clinical tools were applied to our population, our electronic assessment tool had the highest specificity: 0.93 (95% CI 0.88, 0.97) and PPV: 0.54 (95% CI 0.33, 0.74) (Table 4.). The JAMA toolkit had the highest sensitivity: 0.85 (95% CI 0.62, 0.97). All tools had similarly high NPVs.

Table 4

Risk stratification and analysis based on de-labelling tool versus clinician judgement. ^a PPV = positive predictive value. ^b NPV = negative predictive value. Clinician judgement on patient risk determines if patient pursues a skin test or oral challenge. **Supplementary Material**
		Skin test (n)	Oral challenge (n)	Specificity (95%CI)	Sensitivity (95%CI)	PPV^a (95%CI)	NPV^b (95%CI)
PENFAST	≥ 2	8	24	0.85 (0.79, 0.90) 0.66 (0.58, 0.73)	0.40 (0.19, 0.64) 0.85 (0.62, 0.97)	0.25 (0.11, 0.43) 0.24 (0.14, 0.35)	0.92 (0.86, 0.96) 0.97 (0.92, 0.99)
PENFAST	0–1	12	137	0.85 (0.79, 0.90) 0.66 (0.58, 0.73)	0.40 (0.19, 0.64) 0.85 (0.62, 0.97)	0.25 (0.11, 0.43) 0.24 (0.14, 0.35)	0.92 (0.86, 0.96) 0.97 (0.92, 0.99)
JAMA	High – medium	17	55	0.93 (0.88, 0.97) 0.85 (0.79, 0.90)	0.65 (0.41, 0.85) 0.40 (0.19, 0.64)	0.54 (0.33, 0.74) 0.25 (0.11, 0.43)	0.96 (0.91, 0.98) 0.92 (0.86, 0.96)
JAMA	Low	3	106	0.93 (0.88, 0.97) 0.85 (0.79, 0.90)	0.65 (0.41, 0.85) 0.40 (0.19, 0.64)	0.54 (0.33, 0.74) 0.25 (0.11, 0.43)	0.96 (0.91, 0.98) 0.92 (0.86, 0.96)
FIRSTLINE	High – possible	13	11	0.66 (0.58, 0.73)	0.85 (0.62, 0.97)	0.24 (0.14, 0.35)	0.97 (0.92, 0.99)
FIRSTLINE	Very low	7	150	0.66 (0.58, 0.73)	0.85 (0.62, 0.97)	0.24 (0.14, 0.35)	0.97 (0.92, 0.99)

Congruent with previous data, we found low rates of true penicillin allergy in the pregnant population¹⁰. Our study shows that our electronic assessment tool is a reliable electronic tool to safely evaluate and de-label patients similar to that of the gold standard of allergist assessment. The detailed and extensive algorithm likely contributes to its increased specificity and NPV, however, the time required may be more difficult to integrate into a busy practice. The high specificity and NPV of our electronic tool confirms our initial hypothesis and supports its use to DOC patients.

To our knowledge, this is the first study to provide comparison between various available clinical decision support tools for penicillin allergy risk assessment. Our electronic tool has the benefit of both being freely accessible online and providing direct management recommendations. Use of this tool can empower non-allergist healthcare providers to safely identify and manage patients at low risk of true penicillin allergy. This allows for more patients to be de-labelled in community and hospital-settings more quickly and reduces waiting times for allergists to assess and manage patients who are at higher risk of true allergy.

One limitation of this study is there may be recall bias while patients recollected past information for the questionnaire. Additionally, we completed this validation study at a single centre with non-random recruitment, therefore extrapolating data to other populations requires caution.

Highly specific de-labeling tools with high NPV can identify low risk patients while preserving safety. This tool can empower non-allergist healthcare providers to manage low risk patients in a resource limited healthcare system. We herein provide an assessment of reliability and show validity for an electronic point-of-care decision support tool for patients with penicillin allergy. We additionally provide a comparative specificity analysis between our electronic tool and two other published risk-stratification tools in pregnant populations.

intradermal skin testing (IDT)

direct oral challenge (DOC)

BC Women’s Hospital (BCWH)

Positive predictive value (PPV)

Negative predictive value (NPV)

Ethics approval and consent to participate: An ethics and consent waiver from the University of British Columbia was obtained for this quality improvement project.

Consent for publication: not applicable

Availability of data and materials: All clinical decision tools are readily available to public. The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests: Raymond Mak and Tiffany Wong with unrestricted educational grant from ALK. No other conflict of interest from other remaining authors.

Funding: Development of the Firstline penicillin allergy de-labeling tool was supported by the Doctors of BC Physician Quality Improvement Program and the BC Children’s Hospital Medical Staff Association Engagement Funding

Authors’ contributions:

Joanne Wang, Raymond Mak, Tiffany Wong: data collection, drafting, and major edits to the manuscript.

Chelsea Elwood, Vanessa Paquette, Natasha Kwan, Stephanie Erdle, Julie Van Schalkwyk, Ashley Roberts: above authors assisted in manuscript editing.

Melissa Watt: program coordination and data collection

Jeffrey Bone: statistical analysis

Acknowledgements: None

Macfadden DR, Ladelfa A, Leen J, et al. Impact of Reported Beta-Lactam Allergy on Inpatient Outcomes: A Multicenter Prospective Cohort Study. Clinical Infectious Diseases. 2016;63(7):904-910. doi:10.1093/CID/CIW462
Steenvoorden L, Bjoernestad EO, Kvesetmoen TA, Gulsvik AK. De-labelling penicillin allergy in acutely hospitalized patients: a pilot study. BMC Infect Dis. 2021;21(1):1-7. doi:10.1186/S12879-021-06794-1/TABLES/1
Cooper L, Harbour J, Sneddon J, Seaton RA. Safety and efficacy of de-labelling penicillin allergy in adults using direct oral challenge: a systematic review. JAC Antimicrob Resist. 2021;3(1). doi:10.1093/JACAMR/DLAA123
Copaescu AM, Vogrin S, James F, et al. Efficacy of a Clinical Decision Rule to Enable Direct Oral Challenge in Patients With Low-Risk Penicillin Allergy: The PALACE Randomized Clinical Trial. JAMA Intern Med. Published online July 17, 2023. doi:10.1001/JAMAINTERNMED.2023.2986
Zhang BY, Paquette V, McClymont E, et al. Implementing a penicillin allergy delabeling service for the obstetric population. J Allergy Clin Immunol Pract. 2021;9(6):2501-2502.e2. doi:10.1016/J.JAIP.2021.01.023
Roberts H, Soller L, Ng K, et al. First pediatric electronic algorithm to stratify risk of penicillin allergy. Allergy, Asthma and Clinical Immunology. 2020;16(1):1-5. doi:10.1186/S13223-020-00501-6
Shenoy ES, Macy E, Rowe T, Blumenthal KG. Evaluation and Management of Penicillin Allergy: A Review. JAMA - Journal of the American Medical Association. 2019;321(2). doi:10.1001/jama.2018.19283
Trubiano JA, Vogrin S, Chua KYL, et al. Development and Validation of a Penicillin Allergy Clinical Decision Rule. JAMA Intern Med. 2020;180(5). doi:10.1001/jamainternmed.2020.0403
Mak R, Yuan Zhang B, Paquette V, et al. Safety of direct oral challenge to amoxicillin in pregnant patients at a Canadian tertiary hospital. J Allergy Clin Immunol Pract. 2022;10(7):1919-1921.e1. doi:10.1016/J.JAIP.2022.03.025
Blumenthal KG, Shenoy ES. Penicillin Allergy in Pregnancy. JAMA. 2020;323(12):1216-1216. doi:10.1001/JAMA.2019.19809
Mill C, Primeau MN, Medoff E, et al. Assessing the Diagnostic Properties of a Graded Oral Provocation Challenge for the Diagnosis of Immediate and Nonimmediate Reactions to Amoxicillin in Children. JAMA Pediatr. 2016;170(6). doi:10.1001/JAMAPEDIATRICS.2016.0033

SupplementaryMaterial.docx

Download PDF

Journal Publication

published 19 Oct, 2024

Read the published version in Allergy, Asthma & Clinical Immunology →

Reviewers agreed at journal
12 Oct, 2023
Reviewers invited by journal
25 Sep, 2023
Editor assigned by journal
23 Sep, 2023
First submitted to journal
20 Sep, 2023

You are reading this latest preprint version

Reliability and validation of an electronic penicillin allergy risk-assessment tool in a pregnant population

Status:

Journal Publication

Version 1

Abstract

Background:

Methods:

Results:

Conclusion:

Background

Methods

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Supplementary Files

Status:

Journal Publication

Version 1