Detecting Use of Patient Dietary Supplements in Free Text Clinical Notes

doi:10.21203/rs.3.rs-2121730/v1

Download PDF

Research Article

Detecting Use of Patient Dietary Supplements in Free Text Clinical Notes

https://doi.org/10.21203/rs.3.rs-2121730/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: There is widespread use of dietary supplements, some prescribed but many taken without a physician’s guidance. There are many potential interactions of supplements with both over the counter and prescription medications. Many of these are not known to the patient. Documentation of supplement use is incomplete in structured medical records, however additional information about supplements is often found in unstructured clinical notes. Natural language processing (NLP) techniques can be used to detect supplement use in these notes.

Methods: We study a group of 377 patients from three healthcare systems and develop an NLP system to detect supplement use. We then use surveys of these patients to investigate correlation between self-reported supplement use and NLP predictions from the clinical notes.

Results: We attain an F1 score of 0.914 on creation of the model for all supplements. Individual supplement detection had variable correlation with survey responses, ranging from and F1 of 0.83 for calcium, to F1 of 0.39 for folic acid.

Conclusions: We demonstrate the ability to capture the use of dietary supplements from free text clinical notes, enabling clinical studies including drug interactions and outcomes research. Generalizability is demonstrated due to the use of notes from a nationwide electronic health record system. We also show that patients from three healthcare systems self-reported supplement use that often contradicted what was recorded in the clinical record.

Natural Language Processing

Dietary Supplements

Machine Learning

Dietary supplements are widely used in the general population, and many of these supplements are known to interact with prescription medications. Some of these supplements can be dangerous on their own if not taken correctly and can also cause adverse events when combined with other substances.¹ Liu et al. studied surgical evaluations of 376 patients, and found 75% were using complementary therapies, many of which include supplements.² A study by Wood et al. identified the rate of use of complementary therapies at 64%, and that the most frequent were megadose vitamins and nutritional supplements.³ A systematic review of patients with cardiovascular disease found that 36% of those patients took supplements, between 16 and 64% of which were also taking prescription medications.⁴ Many of the known interactions are with prescription cardiac medications, introducing a risk of serious complications.⁵ Loya et al. report that 31.5% of their study participants had a risk of interaction between over the counter medications and supplements they were taking.⁶ In a Veterans Affairs study, Lee et al. found that 61% of their sample of 200 cancer patients used supplements, and 12% of those were at risk of interactions with prescribed medications.⁷ Additionally, a study by Geller et al. found that tens of thousands of emergency room visits every year in the United States are due to adverse events with dietary supplements.⁸

Prevention of adverse events due to these interactions requires knowledge of the medications and supplements being used by a patient. This is normally accomplished through patient medication lists; however, supplements are not routinely present in medication lists unless prescribed by the doctor. Also, as shown by Zhang et al., supplement terminology is standardized between medical terminologies and databases.⁹ Gundlapalli et al. found that supplements can be detected more completely through natural language processing (NLP) analysis of patients unstructured and semi-structured clinical notes.¹⁰ In a large sample, they were able to identify over 28,000 patients using Ginkgo biloba by using clinical notes, where only 9 of them could be identified through structured pharmacy data. Zhang et al. demonstrated the use of rule based and machine learning models for detection of supplement use from clinical notes,¹¹ although the study was performed at a single healthcare system which may limit generalizability.

In this study, we investigate the use of NLP across three different healthcare systems to detect the use of multiple supplements using free-text clinical notes. We then apply the trained model to notes belonging to patients that have completed a survey evaluating their supplement use and evaluate the correlation between patient self-reported supplement use and that detected with NLP techniques in the clinical notes.

Surveys

Paper based surveys were completed by 377 Veterans from three different Veterans Affairs Medical Centers (VAMCs) (130 from Washington DC VAMC, 115 from VA West Haven HCS, and 132 from the VA Salt Lake City HCS) to measure their use of dietary supplements. Informed consent was obtained from each participant after clarification of the study objectives and activities. The characteristics of the participants are shown in Table 1, and a sample of the reported supplements are shown in Table 2. From these surveys a supplement set was created consisting of the unique supplements used by the participants.

Table 1

Characteristics of the survey participants
	Washington DC VA Medical Center	VA West Haven Healthcare System	VA Salt Lake City Healthcare System
Age:	32–88 (61.5)	32–88 (59.1)	24–93 (64.7)
Gender
Female:	20 (15%)	13 (11%)	7 (5%)
Male:	110 (85%)	101 (88%)	125 (95%)
Race
Black:	105 (81%)	41 (36%)	9 (7%)
White:	16 (12%)	67 (58%)	115 (87%)
Other:	1 (1%)	2 (2%)	8 (6%)
Unknown: Ethnicity	8 (6%)	5 (4%)	0 (0%)
Hispanic:	1 (1%)	3 (3%)	9 (7%)
Non-Hispanic:	123 (94%)	107 (93%)	123 (93%)
Unknown:	6 (5%)	5 (4%)	0 (0%)

Table 2

Sample of supplement use from surveys
	Washington DC VA Medical Center	VA West Haven Healthcare System	VA Salt Lake City Healthcare System
Aloe Vera	13	0	0
Black Cohosh	2	0	0
Calcium	0	7	7
Chondroitin	2	8	7
Cinnamon	11	0	2
Coenzyme Q	2	6	7
Cranberry	18	0	0
DHEA	1	1	5
Echinacea	3	2	2
Evening Primrose	1	0	0
Fish Oil	31	13	9
Flaxseed	1	1	1
Folic Acid	10	1	6
Garlic	23	15	7
Ginger	13	1	1
Gingko Biloba	7	6	10
Ginseng	10	6	11
Glucosamine	3	17	10
Goldenseal	3	0	0
Green Tea	41	1	2
Iron	0	3	5
Magnesium	0	8	4
Melatonin	3	15	22
Multivitamins	45	49	49
Potassium	0	2	1
Protein	0	1	5
Saw Palmetto	2	2	2
St Johnś wort	1	1	1
Turmeric	6	1	2
Vitamin A	0	1	3
Vitamin B	27	22	32
Vitamin C	30	9	10
Vitamin D	40	27	39
Vitamin E	21	7	2
Yohimbe	2	0	0
Zinc	0	1	2

Annotation

This study used medical records from the U.S. Veterans Health Administration (VHA) system under IRB supervision through the VA Informatics and Computing Infrastructure (VINCI). VINCI is a collaboration between the VA Office of Information and Technology (OI&T) and the Office of Research and Development (OR&D) to provide researchers with an environment for secure access VA healthcare data.¹² All methods were performed in accordance with the relevant guidelines and regulations. A list of keywords was constructed from the supplement set derived from the surveys and augmented with known variants and misspellings. Keywords were grouped into categories when they had the same meaning. These keywords were used to retrieve a set of clinical notes from patients, not including the survey participants, containing the keywords. These documents were further split into snippets consisting of the matched keyword and its context, +/- 20 words around the keyword. We have found in multiple previous studied that inclusion of 20 words before and after each keyword results in more complete context representation than single sentences, partly due to the non-grammatical structures in many clinical notes which makes automatic sentence splitting unreliable. 1,000 of these snippets were manually reviewed and annotated to indicate if each keyword occurrence indicated current use of supplements (Yes or No).

Machine Learning and Evaluation

A support vector machine (SVM) was trained using features derived from the annotated snippets. Each snippet keyword was considered a separate observation, resulting in 1,913 observations (many of the 1,000 snippets contained multiple keywords). A set of unique 2-grams was constructed from all snippets for use as features. An additional feature was used to indicate the keyword in the snippet, and the outcome was the supplement annotation (Yes or No). The final feature set consisted of 14,217 two-gram features and one keyword feature. 10-fold cross validation was used to measure performance, then all observations were used to create a final model.

Survey Evaluation

To evaluate the ability of the model to identify supplement use in the survey participants, the model was applied to snippets from clinical notes belonging to the participants. Only clinical notes in the time period between one year prior and one month after the survey date were used in order to maintain context with the surveys. Snippets were retrieved and extracted in the same way as in the training of the SVM, using the supplement keywords, resulting in 28,897 snippets. The SVM model was then applied to those snippets to identify if they indicated active supplement use. If the model predicted active supplement use, then the snippet was classified as positive for use of the supplement indicated by the keyword. Snippets were then aggregated to the patient level. A patient was considered to be using the supplement if they had at least one snippet positive for the supplement.

83 keywords representing 44 categories of dietary supplement were identified (Table 3). The SVM trained on the observations derived from the snippets performed with precision = 0.914, recall = 0.914, and f-measure = 0.914 (10-fold cross validation, details in Table 4).

Table 3

Keywords and categories of dietary supplements
keywords	acidophilus, aloe, astragalus, biotin, calcium, cholecalciferol, chondroitin, cinnamon, coenzyme, cranberry, crea, creat, creatine, creatinin, creatinine, creatining, creatnine, cvitamin, cyancobalamin, cyanocobalamin, cyanocobalamine, cyanocobalmin, dhea, digitalis, echinacea, ergocalciferol, fe, ferr, fiber, fish, fishoil, flex, flexion, flx, folic, garlic, ginger, ginkgo, ginko, ginseng, glucosamine, irn, iron, krill, licorice, magnesium, melatonin, mlfolic, multivitamin, multivitamins, multivits, niacin, oregano, palmetto, potasium, potass, potassium, potassuim, pottasium, pyridoxine, retinal, riboflavin, tea, thaimine, thiamin, thiamine, thistle, tumeric, turmeric, vera, viamin, viatmin, vit, vita, vitamin, vitamine, vitd, vitmain, vits, willow, yeast, zinc, zn
categories	aloe vera, astragalus, biotin, calcium, chondroitin, cinnamon, coenzyme Q10, cranberry, creatine, curcumin, dhea, digitalis, echinacea, fiber, fish oil, folic acid, garlic, ginger, ginkgo biloba, ginseng, glucosamine, iron, licorice, magnesium, melatonin, multivitamin, oregano, palmetto, potassium, probiotic, tea, thiamine, thistle, vitamin A, vitamin B12, vitamin B2, vitamin B3, vitamin B6, vitamin C, vitamin D2, vitamin D3, willow, yeast, zinc

Table 4

Performance of SVM model (10-fold cross validation)
Class	Precision	Recall	F1
Yes	0.929	0.928	0.929
No	0.892	0.894	0.893
Weighted Avg.	0.914	0.914	0.914

Agreement of the SVM predictions with supplement use self-reported by patients from surveys was calculated by confusion matrix. Overall, the micro-average precision was 0.79, recall 0.59, f1 0.68. It was observed that roughly half of the supplements had very low representation (occurring in less than 10% of participants). When excluding those rare supplements, overall agreement was similar: micro-average precision was 0.79, recall 0.60, F1 0.68 (Table 5). The best agreement of the well represented supplements, as assessed by F1 measure, was found in calcium (precision 0.81, recall 0.85, F1 0.83), while the worst was found in folic acid (precision 0.70, recall 0.27, F1 0.39).

Table 5

Performance of SVM model compared to patient self-reported supplement use from surveys. Results are shown for supplements reported in > 10% of participants
Supplement Category	Precision	Recall	F1
calcium	0.81	0.84	0.83
chondroitin	0.78	0.64	0.70
creatine	0.77	0.76	0.77
fiber	0.80	0.67	0.73
fish oil	0.79	0.50	0.61
folic acid	0.70	0.27	0.39
iron	0.81	0.68	0.74
magnesium	0.81	0.67	0.73
melatonin	0.68	0.50	0.58
multivitamin	0.91	0.26	0.40
potassium	0.76	0.67	0.71
tea	0.72	0.66	0.69
thiamine	0.78	0.33	0.47
vitamin A	0.77	0.65	0.71
vitamin B12	0.94	0.28	0.43
vitamin D2	0.88	0.47	0.61
vitamin D3	0.94	0.28	0.43
micro-avg	0.79	0.60	0.68
macro-avg	0.80	0.54	0.62

We have demonstrated the ability to capture the use of dietary supplements from free text clinical notes. We trained an SVM model on bi-grams from snippets of text containing supplement keywords, resulting in overall high performance of the model (F1 = 0.914). Health record systems generally contain medications lists, however dietary supplements are not routinely tracked via this mechanism. Some dietary supplements have been shown, and others are strongly suspected, of having interactions with other medications. We have shown that natural language processing (NLP) of semi-structured and unstructured clinical notes can reliably detect the use of many dietary supplements.

We then proceeded to apply our model to the notes of patients that had completed a survey to self-report their supplement use. This was performed on patients at three different health care settings. The ML model agreement with patient self-reported surveys were variable, with good agreement in many cases (e.g. calcium, F1 = 0.83), and needing improvement in others (e.g. folic acid, F1 = 0.39).

A limitation we encountered is the variability in performance between different supplements when comparing to patient self-report in surveys. In error checking our process, we manually reviewed cases where the SVM prediction did not match the survey answers. This uncovered multiple cases where the patient reported that they were not taking a specific supplement, however the supplement was recorded in their medication list. There was an apparent pattern indicating that patients did not regard something as a supplement if it was prescribed by a physician. This may be the underlying cause of lower agreement of our model in cases such as folic acid and melatonin, both of which are commonly prescribed by a physician. Future studies could address the mental model patients have of what they consider to be a supplement. Also, roughly half of the supplements we studied did not have the number of observations needed for reliable results to be measured when compared to surveys. A larger survey set, or more directed survey, would be useful to address this. Another limitation is that we did not perform feature selection or parameter optimization before training the SVM, nor did we compare SVM against other ML algorithms. Although SVMs perform their own internal feature selection, better optimization may be obtained from feature selection and parameter tuning.

NLP

Natural language processing

VAMC

Veterans Affairs Medical Center

HCS

Healthcare System

VHA

Veterans Health Administration

VINCI

VA Informatics and Computing Infrastructure

OI&T

Office of Information and Technology

OR&D

Office of Research and Development

SVM

support vector machine

Machine Learning

Ethics approval and informed consent to participate

Approval was obtained from the institutional review boards and the research and development committees of the Washington DC VAMC, VA West Haven HCS, and the VA Salt Lake City.

All subjects were properly consented to according to the approved protocol.

Consent for publication

Not applicable.

Availability of data and materials

The datasets generated during and/or analyzed during the current study are not publicly available to protect the privacy of research participants but aggregated datasets are available from the corresponding author on reasonable.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was supported by The Clinical and Translational Science Institute at Children’s National (CTSI-CN) through the National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) grant numbers UL1TR001876 and KL2TR001877.

Support was also provided by the Veterans Health Administration grant PPO 14-382

Authors’ contributions

DR carried out the NLP development and evaluation and drafted the manuscript, EW participated in NLP, YS and YC participated in the data analysis, JHG and CB contributed to the study design, and QZT conceived the study and oversaw the manuscript preparation. All authors have reviewed the manuscript and contributed to the writing.

Acknowledgments

N/A

Cupp MJ. Herbal remedies: Adverse effects and drug interactions. Am Fam Physician. 1999;59(5):1239–44.
Liu EH, Turner LM, Lin SX, et al. Use of alternative medicine by patients undergoing cardiac surgery. J Thorac Cardiov Sur. 2000;120(2):335–41.
Wood MJ, Stewart RL, Merry H, Johnstone DE, Cox JL. Use of complementary and alternative medical therapies in patients with cardiovascular disease. Am Heart J. 2003;145(5):806–12.
Grant SJ, Bin YS, Kiat H, Chang DHT. The use of complementary and alternative medicine by people with cardiovascular disease: a systematic review. Bmc Public Health. 2012;12.
Tachjian A, Maria V, Jahangir A. Use of Herbal Products and Potential Interactions in Patients With Cardiovascular Diseases. J Am Coll Cardiol. 2010;55(6):515–25.
Loya AM, Gonzalez-Stuart A, Rivera JO. Prevalence of Polypharmacy, Polyherbacy, Nutritional Supplement Use and Potential Product Interactions among Older Adults Living on the United States-Mexico Border A Descriptive, Questionnaire-Based Study. Drug Aging. 2009;26(5):423–36.
Lee AH, Ingraham SE, Kopp M, Foraida MI, Jazieh AR. The incidence of potential interactions between dietary supplements and prescription medications in cancer patients at a Veterans Administration Hospital. Am J Clin Oncol-Canc. 2006;29(2):178–82.
Geller AI, Shehab N, Weidle NJ, et al. Emergency Department Visits for Adverse Events Related to Dietary Supplements. New Engl J Med. 2015;373(16):1531–40.
Zhang R, Manohar N, Arsoniadis E, et al. Evaluating Term Coverage of Herbal and Dietary Supplements in Electronic Health Records. AMIA Annu Symp Proc. 2015;2015:1361–1370.
Gundlapalli AV, Redd D, Gibson BS, et al. Maximizing clinical cohort size using free text queries. Comput Biol Med. 2015;60:1–7.
Fan YD, He L, Zhang R. Evaluating Automatic Methods to Extract Patients' Supplement Use from Clinical Reports. Ieee Int C Bioinform. 2017:1258–1261.
VA Informatics and Computing Infrastructure (VINCI). Veterans Health Administration Web site. https://www.hsrd.research.va.gov/for_researchers/vinci/. Accessed2019.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Detecting Use of Patient Dietary Supplements in Free Text Clinical Notes

Status:

Version 1

Abstract

Introduction

Materials And Methods

Surveys

Annotation

Machine Learning and Evaluation

Survey Evaluation

Results

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1