A Comparison Of Text Mining Versus Diagnostic Codes To Identify Opioid Use Problem: A Retrospective Study​

doi:10.21203/rs.3.rs-16124/v1

Download PDF

Research article

A Comparison Of Text Mining Versus Diagnostic Codes To Identify Opioid Use Problem: A Retrospective Study

https://doi.org/10.21203/rs.3.rs-16124/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background As opioid prescriptions have risen, there has also been a rise in opioid overdose deaths and substance use disorders. Public health systems have tried to improve their ability to detect and intervene in opioid use disorders to prevent death due to overdose. The objective of this study is to compare two approaches to identify opioid use problems (OUP) using electronic health record data- text mining versus diagnostic codes.

Methods Our sample consisted of adults on long-term opioid therapy (LTOT), defined as at least ≥ 70 days of supply within 90 days, and who visited a large multi-hospital network within a two-year period, between 1 January 2013 and 31 December 2014. We excluded patients with active cancer or schizophrenia. Text mining results were validated by a semi-assisted human review process and positive predictive value and level of agreement was reported. Each algorithm sought to identify patients who visited a health care facility due to an opioid poisoning event, opioid abuse, or opioid dependence. Population characteristics for positive OUP identified by text mining and ICD cohorts were compared. Chi-square and Fishers exact test were used for categorical data analysis and independent t-test was used to compare means for continuous variables. We further compared the demographics of the cohorts identified by the two methods.

Results We identified 14,298 eligible LTOT patients. Text mining of relevant electronic clinical notes yielded 127 positive OUP cases compared to 45 cases using International Classification of Disease (ICD)-9 codes for the same population. Just eight OUP patients were identified using both methods. The two cohorts differed significantly with respect to age, gender, and other characteristics.

Conclusions Compared to diagnostic codes, text mining identified more OUP cases with distinct characteristics. Incorporating text-mining techniques into OUP surveillance methods may support better detection of OUP and more accurate estimates of prevalence.

Medical Informatics

Opioid Abuse

Diagnosis

Clinical Decision making

Text mining

Natural language processing

Opioids are a class of medications prescribed to relieve pain. Before the 1980s, opioids were mainly prescribed by surgeons to relieve immediate postoperative pain or pain related to cancer (1, 2). However, during the 1980s, aggressive pain treatment using opioids for chronic, non-cancer pain was encouraged(1, 3). Advocacy groups, such as the American Pain Foundation, continued to push for more aggressive pain management including more liberal use of opioids during the late 1990s and through the 2000s (1, 4–6). Opioid prescriptions increased from 142 to 248 million between 1999 and 2012 (7, 8), and opioid sales quadrupled from 1999 to 2010 (9). Specific medications, such as hydrocodone, doubled in consumption between 1999 and 2011 (10).

In addition to an increase in the number of opioid prescriptions, the percentage of opioid overdose deaths and substance use disorder treatment increased from 1999 to 2008 (11). In 2015, drug overdose incidents accounted for 52,404 U.S. deaths and opioid use was involved in approximately 63% of those incidents (12). The 2016 National Survey on Drug Use and Health indicated that there were approximately 11.8 million people in the U.S. aged 12 years or older who misused opioids in the past year (13). At the state level, Indiana reported 785 opioid-related overdose deaths with a substantial increase in heroin-related overdose deaths (14). Compared to 2012, heroin-related overdose deaths increased from 110 to 296 in 2016 (14).

Despite opioid misuse becoming an epidemic and receiving a public health emergency declaration by the U.S. government in 2018, reporting of opioid problems may still impose some challenges and is likely underreported. For example, differences in target outcome definitions (“opioid use disorder” vs. “opioid overdose” vs. “opioid abuse”) and population inclusion criteria (minimum medication day’s supply or number of opioid prescriptions) may alter the ability to achieve consensus on the problem definition and the population that requires intervention (15). Despite these differences, the International Classification of Diseases (ICD) is a standard method used by health care providers to diagnose opioid use disorder, overdose, abuse, and dependence (which we will refer to in this study as Opioid Use Problems or OUP). However, the issue with reporting opioid use problems defined by ICD codes is the fact that these codes often understate the actual number of patients exhibiting the target categorization (16, 17).

Thus, we examined the literature to identify alternative methods to identify OUP in addition to ICD codes. We found a limited number of studies that discuss alternative methods such as natural language processing (NLP), or text mining, in a general healthcare setting (18–21). The use of text mining in these studies varied from accelerating documentation in patient records to automatically parsing and coding clinical events or assisting clinical decision-making by classifying specific complex diagnoses.

More relevant studies addressing NLP/text mining to identify OUP were identified as well (22–24). Carrell et al. (2015) developed an NLP process to identify OUP in electronic health records (EHRs) at Group Health, a healthcare system in Seattle, WA, USA. The study found that conventional diagnostic codes for OUP identified 2,240 (10.1%) patients and NLP identified an additional 728 (3.1%) patients by analyzing patients’ clinical notes (24). Hylan et al. (2015) also used NLP to “report on a predictive model developed to assess the likelihood of problem opioid use over a 2- to 5-year period in patients on chronic opioid therapy” within the Group Health system. For the regression model predicting problem opioid use, the sensitivity was e 58.3% and specificity 71.2% (22). While these early efforts to use NLP/text mining are encouraging, all identified studies were developed and evaluated in a single health care organization, limiting the generalizability. Thus, in our study, we sought to examine the use of NLP on identification of OUP across different populations and healthcare settings and compare to the use of ICD-9 codes in those same settings.

In this study, we employed methods for identifying OUP using administrative and EHR data and reported identified patients’ characteristics and healthcare utilization for both methods. We discuss how identified populations differ and overlap, and whether adding a text mining approach could improve identification of OUP for patients on long-term opioid therapy (LTOT) for a large population across a multi-site healthcare system.

a. Methods Overview

Following Institutional Review Board (IRB#: 1710876219) and Regenstrief Institute (RI) Data Management Committee approvals, we identified an LTOT cohort pulled from clinical notes within Indiana University Health (IUH), a large healthcare network in Indiana with 3,541 staffed beds and 2,563,086 outpatient visits across 18 facilities (25). OUP was determined using diagnostic codes from patients’ health records and a text-mining algorithm was applied to the clinical notes. Finally, we compared the two approaches using frequency and rate of OUP per 1,000 LTOT patients. Additionally, we compared health care utilization (outpatient visits, emergency department visits, hospitalizations, cumulative hospitalization days).

b. Sample Definition

Our sample consisted of adult (age >= 18 years) patients who visited an IUH facility and received LTOT, which we defined as patients prescribed 70 days of supply within any given 90-day period, between 1 January 2013 and 31 December 2014 (24 months). We excluded patients with active cancer (ICD-9 codes 140.x 172.x, 174.x 209.xx, 235.x 239.xx, 338.3) to focus on LTOT for non-cancer chronic pain. Patients with schizophrenia were also excluded due to the documented high percentage of opioid dependence among this population (ICD-9 code 295.9) (26).

c. Variables of Interest

We compared patient characteristics of interest based on the OUP phenotype and included: demographics (age, gender, race, and ethnicity), alcohol abuse, non-opioid abuse, tobacco use, mental health disorders, and hepatitis C. For this study, mental health disorders were identified as depressive disorder (ICD-9 codes 296.2x, 296.3x, 300.4, 311), suicide attempt or other self-injury (ICD-9 codes E95x.x, E98x.x), or anxiety disorder (ICD9 codes 300.0x, 300.21, 300.22, 300.23, 300.3, 308.3, 309.81).

d. Report Type Selection

Due to the large number of clinical report types generated and included in the EHR, we included only report types that were deemed most relevant to clinical and data experts to OUP identification. Nine report types were selected as most relevant for OUP identification. The query returned 142,971 reports: Emergency Department Doctor Progress Notes (48,898), Emergency Department Discharge Notes (28,637), Primary Care Doctor Outpatient Progress Notes (26,669), Visit Notes (21,868), Discharge Summaries (11,731), History and Physicals (2,759), Admission History & Physicals (1,390), Preadmission History/Physicals (576), Primary Care Doctor Outpatient History and Physical /Initial Consulta (443). The top 5 most common report types generated over 96% (137,802) of total reports, while the bottom 4 least common were only responsible for 4% of reports generated. Due to labor constraints, the bottom 4 report types (Admission History and Physical, Preadmission History/Physical, Primary Care Doctor Outpatient History and Physical /Initial Consult) were excluded.

e. Identify Patients with OUP

We used two methods to identify OUP: 1) text-mining and 2) ICD-9 approaches.

I. Identify OUP using a Text Mining Process

The process of identifying OUP using text mining involved 2 main steps: 1) algorithm development to flag potential positive reports; and 2) validation using semi-assisted manual review.

1- Algorithm Development to Flag Potential Positive Cases Using nDepth™

We applied the text-mining package provided by nDepth™ to parse medical notes to detect OUP. nDepth™ is an NLP tool designed by the Regenstrief Institute in Indianapolis to extract data from the Indiana Network for Patient Care (INPC), which is a healthcare database managed by the Regenstrief Institute on behalf of the Indiana Health Information Exchange (IHIE) (27, 28). As IUH is part of INPC, nDepth™ was used to query patients’ clinical notes to identify possible opioid use problems. To develop the algorithm, 2 keyword lists adopted from the literature were entered into nDepth™ (24). The first list was comprised of opioid terms (e.g., Vicodin, Opiate), and the second list was comprised of problem terms (e.g., addiction, abuse) (Appendix A.1). nDepth™ creates state machines, an algorithm that parses report types for certain criteria, to check for all possible combinations of the 2 lists within a 5-word distance. Flagged results are automatically checked for whether the statement is negated, hypothetical, historical, or experienced. This process itself was adopted from the ConText algorithm developed by Harkema et al. (2009) (29). Thus, if the system determined the patient statement was not negated or deemed hypothetical or historical, those flags were deemed as experienced (considered positive).

2- Semi-Assisted Manual Review Validation Process

Flagged reports were reviewed by 2 trained reviewers who used a semi-assisted manual review. In case of disagreement, an expert physician acted as a third reviewer to resolve the dispute. To avoid overlooking any signs of opioid use problems, nDepth™ was programmed to highlight 27 suggestive phrases in the flagged reports. These phrases were collected from the literature and modified based on common clinical dialog in Indiana (Appendix A.2). In cases where there was more than one flagged report per patient, the system randomly selected one type of the flagged reports to be reviewed per patient. The criteria chosen to determine OUP were adopted from a study by Carrell et al. (2015) and listed in Table 1. Of note, as using marijuana medically and recreationally is currently illegal in Indiana, its use was treated as concurrent use of an illicit drug during manual review. Other modifications were also adopted based on initial reviews of subsets of patients’ clinical reports.

Table 1 The criteria for identifying opioid use problems in patients’ clinical notes

No.	Criteria for opioid use problems
1	Substance abuse treatment, including referral or recommendation
2	Methadone or suboxone treatment for addiction
3	Obtained opioids from nonmedical sources
4	Loss of control of opioids, craving
5	Family member reported patient’s opioid addiction to clinician
6	Significant treatment contract violation
7	Concurrent alcohol abuse/dependence (not remitted)
8	Concurrent use of illicit drugs
9	Current or recent opioid overdose
10	Pattern of early refills (not an isolated event)
11	Manipulation of physician to obtain opioids
12	Obtained opioids from multiple physicians surreptitiously
13	Opioid taper/wean due to problems, lack of efficacy (not due to expected pain improvement)
14	Unsuccessful taper attempt
15	Rebound headache related to chronic opioid use
16	Concurrent use of unauthorized narcotics (polypharmacy)
17	Physician states opioid abuse/overuse/addiction or listed ICD codes for opioid abuse/dependence

II. Identify OUP Using ICD-9 Codes

Two definitions from the literature utilizing ICD-9 codes were combined to create a case definition of OUP: opioid abuse and dependence (304.00, 304.01, 305.50, 305.51, 304.71, 304.02, 304.70, 305.52) and opioid poisoning (965.0, 965.00, 965.01, 965.02, 965.09, E850.0, E850.1, E850.2) (Appendix A.3). These definitions were combined to minimize the chance of systematically creating type II errors by capturing the wider spectrum of opioid use problems among the study population.

f. Analysis

Frequency tables are used to describe population characteristics. Population characteristics for positive OUP identified by text mining and ICD cohorts were compared. Chi-squared and Fishers exact tests were used for categorical data analysis and independent t-tests were used to compare means for continuous variables.

We identified and compared OUP using ICD-9 codes and text mining techniques.

The results are summarized in Figure 1.

a. Sampling Results

Adult patients from IUH on LTOT were queried and our inclusion criteria returned 34,661 patients. Applying our exclusion criteria, cancer and schizophrenia, excluded 11,579 (33.4%) patients (11,121 cancer patients, 272 with schizophrenia [186 with both schizophrenia and cancer]). We identified 23,082 patients who received LTOT and generated 559,464 reports across the IUH system. Our report selection criteria excluded 8,784 patients, leaving 14,298 eligible patients which generated 137,804 reports. Nearly 60% of the LTOT population were over 55 years old, and women represented 62% of the LTOT population. The white race was in the majority among the LTOT population (86%), while black patients represented 12%, and other or unknown races were 2%. Non-Hispanic or Latino ethnicity represented 87% of the LTOT population (Table 2).

b. Identification of Opioid Use Problems Using a Text-Mining Approach

Our algorithm flagged 468 distinct statements in 366 unique reports representing 154 patients. For validation and to confirm the presence of OUP, 2 reviewers each reviewed 1 flagged report per positive patient based on specific criteria (Table 1). Out of 154 flagged patients, only 127 patients were deemed as positive cases of OUP with a positive predictive value (PPV) of 82%. Cohen’s κ was run to determine if there was agreement between the 2 reviewers’ judgement on whether a subset of 200 patients in the study cohort were meeting any OUP criteria. There was moderate agreement between the 2 reviewers’ judgements: κ = 0.691 (95% CI, 0.58 to 0.79), p < .0005.

Table 2 Characteristics of individuals on long term opioid therapy exclusive of those with cancer or schizophrenia

Demographics	Study Cohort
Demographics	N(%)
Age (18-24)	236 (1.7%)
25-34	891 (6.2%)
35-44	1,749 (12.2%)
45-54	2,808 (19.6%)
55-64	3,222 (22.5%)
>65	5,392 (37.7%)
Sex (women)	8,923 (62.4%)
Men	5,375 (37.6%)
Race (Black)	1,719 (12.0%)
White	12,367 (86.5%)
Other/Unknown	212 (2.0%)
Ethnicity (Not Hispanic or Latino)	12,369 (86.5%)
Hispanic or Latino	138 (1.0%)
Unknown	1,791 (12.5%)
Alcohol Abuse (Yes)	88 (0.6%)
Non-opioid Abuse (Yes)	88 (0.6%)
Tobacco Use (Yes)	912 (6.4%)
Depression (Yes)	845 (5.9%)
Self-injury (Yes)	14 (0.1%)
Hepatitis C (Yes)	94 (0.7%)
Total	14,298

Frequency distribution of criteria to identify OUP in patients’ clinical notes were ranked as follows [OUP criteria, Frequency (percentage)]: [Physician states narcotic abuse/overuse/addiction or listed ICD codes for opioid abuse/dependence, 48 (38%)]; [Concurrent use of illicit drugs, 18 (14%); Methadone or suboxone treatment for addiction, 15 (12%)]; [Concurrent use of unauthorized narcotics (polypharmacy), 11 (9%)]; [Current or recent opioid overdose, 10 (8%)]; [Concurrent alcohol abuse/dependence (not remitted), 9 (7%)]; [Obtained opioids from multiple physicians surreptitiously, 8 (6%)]; [Physician or patient wants immediate taper, 3 (2%)]; [Substance abuse treatment, including referral or recommendation, 2 (2%)], [Significant treatment contract violation, 2 (2%)]; [Family member reported patient’s opioid addiction to clinician, 1 (1%)]

c. Text Mining OUP Identification in Comparison To ICD-9

Querying the same LTOT population for designated ICD-9 codes to identify OUP returned 49 distinct diagnoses representing 45 unique patients (4 patients had 2 distinct positive OUP ICD-9 codes). The frequency distribution of positive ICD-9 codes for OUP were ranked as the following [ICD-9 code, Frequency (percentage)]: [304 “opioid dependence unspecified”, 24 (49%)], [305.5 “Nondependent opioid abuse”, 9 (18%)], [304.01 “opioid dependence continuous”, 5 (10%)], [965 “Non-specific-Poisoning by opiates and related narcotics”, 3 (6%), 965.01, 3 (6%)], [E850.2 “Accidental poisoning by other opiates and related narcotics”, 2 (4%)], [304.71 “opioid/other dependence continuous”, 1 (2%)], [965.09 “Specific -Poisoning by other opiates and related narcotics”, 1 (2%)], [E850.0 “Accidental poisoning by heroin”, 1 (2%)] (Appendix A.4). Eight patients were identified as having OUP using both the text mining approach and ICD-9 codes (Appendix A.5).

The frequency distribution of OUP positive cohorts’ characteristics is summarized in Table 3. Women had a higher frequency of OUP among ICD cohorts compared to the text mining cohort (71% and 48%). However, Figure 2 indicates that men had a higher rate of OUP than women (14.7 vs. 10.4 per 1,000).

The frequency distribution stratified by age group shows that the 45-54 age group had the highest frequency of OUP (31% and 24%) for both the ICD and text mining cohorts. The 18-24 age group had the highest rate of OUP among age groups in ICD and text mining cohorts (12.7 and 42.4 per 1,000) (Figure 2).

Table 3 Demographic characteristics of ICD-9 and Text mining OUP cohorts

Characteristic	Study Cohorts*			Significance
Characteristic	ICD-9 N (%)	Text-Mining N (%)	Combined N (%)	X² (P-value)**
Age (18-24)	3 (6.7%)	10 (7.9%)	13 (7.9%)	10.2 (< 0.001) †
25-34	7 (15.6%)	22 (17.3%)	26 (15.9%)
35-44	4 (8.9%)	29 (22.8%)	33 (20.1%)
45-54	14 (31.1%)	31 (24.4%)	42 (25.6%)
55-64	7 (15.6%)	25 (19.7%)	31 (18.9%)
>65	10 (22.2%)	10 (7.9%)	19 (11.6%)
Sex (Women)	32 (71.1%)	61 (48.0%)	89 (54.3%)	7.1 (0.0076) †
Men	13 (28.9%)	66 (52.0%)	75 (45.7%)
Race (Black)	5 (11.1%)	23 (18.1%)	27 (16.5%)	1.6 (0.6014)
White	39 (87.7%)	101 (79.5%)	134 (81.7%)
Other/Unknown	1 (2.0%)	3 (2.4%)	3 (1.8%)
Ethnicity (Not Hispanic or Latino)	39 (86.7%)	106 (83.5%)	139 (84.8%)	0.8 (1)
Hispanic or Latino	0 (0.0%)	2 (1.6%)	2 (1.2%)
Unknown	6 (13.3%)	19 (15.0%)	23 (14.0%)
Other Characteristics Alcohol Abuse (Yes)	2 (4.4%)	3 (2.4%)	5 (3.0%)	0.5 (0.6069)
Non-opioid Abuse (Yes)	10 (22.2%)	7 (5.5%)	14 (8.5%)	10.4 (0.0028) †
Tobacco Use (Yes)	14 (31.1%)	17 (13.4%)	30 (18.3%)	7 (0.0079)
Depression (Yes)	12 (26.7%)	14 (11.0%)	23 (14.0%)	6.3 (0.0118) †
Self-injury (Yes)	2 (4.4%)	2 (1.6%)	3 (1.8%)	1.2 (0.2804)
Hepatitis C (Yes)	3 (6.7%)	3 (2.4%)	5 (3.0%)	1.8 (0.1848)
Total Cohorts	45	127	164*	—
ICD-9: Positive OUP cases using ICD-9 codes; Text Mining: Positive OUP cases using a text mining approach; Combined: Combined positive cases in ICD-9 codes or a text mining approach (There are 8 cases that overlapped between ICD and text mining cohorts); (%): Column percentage. * Chi-square was reported for 22 measurements and Fishers exact test (Freeman-Halton test) for r x c tables. † Significant difference between ICD and Text mining cohort on α = 0.05.

Similar to the overall study population, race and ethnicity were most frequently White, non-Hispanic in both positive cohorts (Table 3). Other characteristics associated with OUP in the literature were reported, including alcohol abuse, non-opioid abuse, tobacco use, depression, self-injury, and hepatitis C. All showed a pattern of being at their lower point in the overall study cohort, with a modest increase in the positive text mining cohort and peaking at the positive ICD-9 cohort (Table 2 and Table 3). An independent-samples t-test was conducted to compare care utilization (outpatient visits, emergency department visit, hospitalizations, and cumulative hospitalization days) among OUP cohorts (Table 4). Our analysis shows that the text mining cohort had significantly higher average of visiting emergency department (M = 3.17, SD= 4.9) compared to ICD cohort (M = 1.9, SD= 2.1), P = 0.022. The ICD cohort had a significantly higher average of cumulative hospitalization days (M = 16.2, SD= 23.9) compared to the text mining cohort (M = 7.4, SD= 12, P = 0.022.)

Table 4 Comparison of care utilization among OUP cohorts

Care Utilization	ICD cohort Mean ± (SD)	Text mining cohort Mean ± (SD)	P-value
Outpatient Visits	12.84 ±(15.05)	12.87 ± (12.402)	0.992
Emergency Department visits	1.91 ± (2.193)	3.17 ± (4.915)	0.022 †
Hospitalizations	2.24 ± (2.838)	1.55 ± (2.572)	0.154
Cumulative Hospitalizations Days	16.24 ±(23.985)	7.43± (12.016)	0.022 †
† Results are significant on α = 0.05

d. Performance of Variables to Identify OUP in Text Mining population

We conducted bi-variate analyses for variables of interest to compare positive vs. negative OUP among text mining cohorts in the study population (Table 5). The following variables were statistically significant (alpha= 0.05) and were ranked based on likelihood ratios and approximate change in probability: Non-opioid Abuse, Gender, Tobacco Use, Self-Injury, Depression, Alcohol, Abuse, and Hepatitis C. T-tests were also significant for the following continuous variables: age, outpatient visits, emergency department visits, hospitalizations, cumulative hospitalization days.

Table 5 Measure of association and likelihood ratio for categorical variables among text mining cohort

Variable	X²	p-value	Likelihood ratio	p-value	Approximate change in probability
Non-opioid Abuse	55.457	< 0.001	20.083	< 0.001	>45%
Gender	11.23	0.001	10.849	0.001	>45%
Tobacco Use	10.881	0.001	8.41	0.004	>40%
Self-Injury	30.971	< 0.001	7.947	0.005	>35%
Depression	6.186	0.013	4.994	0.025	>25%
Alcohol Abuse	6.616	0.01	3.823	0.051	>20%
Hepatitis C	5.895	0.015	3.517	0.061	>20%

Table 6 Comparison of continuous variables among positive OUP using text mining vs. negative cohorts

Variable	Text mining positive	Mean	Std. Deviation	p-value
Age	Yes	44.98	13.24	< 0.001
	No	59.02	16.34
Outpatient Visits	Yes	12.86	12.4	0.001
	No	9.15	10.48
Emergency Department visits	Yes	3.17	4.91	< 0.001
Emergency Department visits	No	0.94	1.86
Hospitalizations	Yes	1.55	2.57	< 0.001
	No	0.54	1.138
Cumulative Hospitalization Days	Yes	7.42	12.01	< 0.001
Cumulative Hospitalization Days	No	3.21	17

a. Identification of Opioid Use Problems

We measured the prevalence of opioid use problems in IUH adult patients using text mining and ICD-9 codes. In our study, a text mining approach identified 127 (0.8%) OUP cases out of 14,298 patients, while ICD-9 identified 45 (0.3%) OUP cases from the same population (Total combined = 164 [1.1%]). Carrell et al. (2015) identified 1,875 (8.5%) patients out of 21,795 patients eligible for the study using NLP methods, while ICD-9 identified 2,240 (10.1%) from the same population. We believe the difference of OUP prevalence compared to prior studies could be due to several factors such as the difference between opioid problem rates in populations or study design and identification criteria. In our study, we included report types that pertain to emergency department and primary care visits, as opposed to the Carrell study that included behavioral\mental health reports protected under 42 CFR part 2 - confidentiality of substance use disorder patient records (30). These reports would likely include OUP-related data that would increase our ability to detect OUP prevalence similar to the prior work.

b. Frequency of ICD Criteria to Identify OUP

Overall, our study population generated 49 ICD-9 events, representing 45 distinct patients. The most commonly used ICD-9 codes were 304 [opioid dependence unspecified] (49%), 305.5 [opioid abuse unspecified] (18%), and 304.1 [opioid dependence continuous] (10%). These findings are similar to the results reported by Palmer et al. (2015) which investigated the prevalence of opioid use problems among chronic opioid therapy patients in the Group Health Cooperative from 2006 to 2012. The most commonly reported ICD-9 codes from that study were 304 [opioid dependence unspecified] (55%), 304.1 [opioid dependence continuous] (26%), and 305.5 [opioid abuse unspecified] (10%) (24). The pattern of documenting OUP using unspecified ICD codes was common and a similar pattern was noticed by our reviewers during the manual review process of patients’ clinical notes. In this study, we found unspecified criteria [Physician states opioid abuse/overuse/addiction or mentions unspecified ICD codes for opioid abuse/dependence] represented 38% of positive OUP in the text mining approach. These findings suggest a common pattern in OUP identification between ICD-9 codes and a text mining approach.

c. Frequency of Text Mining Criteria to Identify OUP

We used 17 criteria to identify OUP from clinical notes. The main differences in these criteria compared to prior studies are: 1- we included marijuana in our list of illicit drugs 2- We added concurrent use of authorized opioids to include specific cases of polypharmacy of opioid abuse 3- We added a criteria to cover cases where physician states “narcotic” abuse, overuse, addiction in the medical record or listed ICD codes for opioid abuse/dependence. In our study, use of an illicit drug identified 18 cases, of which 3 cases indicated active marijuana use (within the last month) without mentioning other illicit drug abuse. Polypharmacy of opioid abuse flagged another 11 OUP cases. The third criteria yielded 48 positive OUP cases, of which, 16 cases have one or more mentions of ICD-9 code diagnosis of abuse and dependence within the clinical notes. Out of those 16 patients, there was only one patient who had a documented ICD-9 for opioid abuse and dependence in the structured data.

d. Demographics

In this study, we found women were represented more than men, overall. This is consistent with Carrell et al. (2015) and Hylan et al. (2015) which both found women to represent about two-thirds of the chronic opioid therapy population in their studies. Overall, higher women’s representation is consistent with the notion that women are more likely than men to use psychotropic and opioid medications (31-33). However, men generally have higher rate of opioid overdose deaths per 100,000 at the national level (20 vs. 9) as well as at Indiana state level (23 vs. 12) (34) . Our study results are consistent with this notion; while OUP distribution stratified by gender showed a higher rate of women with OUP, the rate of overall OUP was higher among men. In our study, OUP among men were more frequently identified using text mining (52%) vs. ICD (29%) (Table 3). This variation could be attributed to the text mining identification criteria itself or it could be attributed to other factors such as gender bias in psychotropic drugs and addiction treatment (35, 36). Studying the root cause of less documentation of OUP among men using ICD codes may be an effort for future research.

A Note about Opioid Problem Detection Using Structured Data

Despite the study which used a combination of 2 types of ICD-9 codes (opioid abuse and dependence and poisoning), overall, our text mining approach identified more cases. The text mining approach also detected 15 patients with a mention of an ICD-9 code of opioid abuse and dependence in patients’ clinical notes, but these codes were undocumented in patients’ record as structured data. This might indicate a significant lack of documentation toward opioid use problems and confirms previous studies’ findings regarding the under-reporting of OUP using ICD-9. For future research pertaining to identifying OUP, we recommend using alternate methods (in addition to ICD codes), such as text mining and machine learning, and exploring other commonly documented structured data, such as procedural codes and electronic lab results.

e. Limitations

In this study, we used a text mining approach to analyze patients’ clinical notes to identify OUP and compared to using ICD-9 codes to identify OUP. However, the analysis was limited to 5 report types. Developing and implementing NLP techniques generally requires intensive computing and an initial commitment of substantial resources and expertise (37). Thus, it was not financially feasible within the allocated budget for this study to test all report types generated by the IUH study population. Rather, the authors relied on expert opinions to meaningfully limit the number of report types. Reliance on expert opinion during algorithm development is common in opioid problem identification literature. Canan et al. (2017) reviewed 15 automated algorithms to identify nonmedical opioid use using electronic health record data. The study found that investigators explicitly relied on subject matter experts during the process of algorithm development and to identify candidate variables (37).

Another limitation pertaining to study results is that, despite text mining correctly identifying 127 OUP cases out of 154 initial possible cases (PPV = 82%), the overlap between the text mining cohorts and ICD-9 cohorts was only 8 cases (17%) (Appendix A.5). This may suggest that the text mining method has missed some positive cases (false negative). To investigate this assumption, we have reviewed 500 randomly selected reports that were not flagged by our text mining criteria. Among the 500 reports, we identified 10 false negative cases, indicating a negative predictive value of 98% on the reviewed subset. Moreover, it is important to emphasize that our study did not have access to behavioral\mental health reports. These reports may contain additional information on OUP which could increase the overlap between the two identification methods.

Finally, we used multiple behavioral concepts to define opioid use problems (dependence, abuse, and addiction) and thus, distinguishing specific behavioral concepts was not automated in this research. Future work may investigate using text mining, natural language processing, and machine learning to better target specific behaviors.

In this study, we developed a text mining approach to identify OUP among patients on long-term opioid therapy at IUH. Compared to ICD-9 codes, text mining successfully identified more OUP cases within the study population. The relatively small overlap between cohorts identified by ICD and text mining may suggest that it may be better to use a combination of both techniques to identify a wider spectrum of OUP. Future development of text mining techniques can help identify cases undiscovered by conventional ICD-9 reporting methods.

ICD

International classification of disease

OUP

Opioid use problems

EHR

Electronic health record

NLP

Natural language processing

LTOT

long-term opioid therapy

Regenstrief Institute

IUH

Indiana University Health

INPC

Network for Patient Care

IHIE

Indiana Health Information Exchange

PPV

Positive Predictive Value

Ethics and Consent statement

This study received ethics approval from the Institutional Review Board at Indiana University (Study No. 1710876219). The IRB further granted the study a waiver of authorization for participation due to the retrospective design using data already collected by the public health agency.

Consent to Publish

Not applicable

Availability of data and materials

The data was collected from Indiana University Health through the Regenstrief Institute. The data was obtained through a data use agreement. These data cannot be shared with other scientists by the authors as they contain protected health information, and sharing the data would constitute a breach of the agreement.

Competing interests

The authors declare that they have no competing interests.

Funding

This project was supported by Deanship of Scientific Research and Research Center, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia. This work was further supported by the Trustees of Indiana University, Bloomington, Indiana, USA (Grant PI: Newhouse, Project PI: Embi).

Authors' Contributions

Several authors (AHZ, BED, MJB, and JFJ) contributed to the conception and design of the study. The primary author (AHZ) drafted the manuscript, and XL and LAA supported the data analysis. Two authors (AHZ and JSP) reviewed patients’ clinical notes. Two authors (BED and MJB) provided insight on study design and results interpretations. All authors reviewed, commented, and participated in revisions of this manuscript. All authors read and approved the final manuscript.

Acknowledgments

This study involved a research collaboration among Indiana University’s School of Medicine, the IU Fairbanks School of Public Health, the School of Informatics and Computing at IUPUI, and the Regenstrief Institute. The authors thank the analysts at the Regenstrief Institute Data Core for their assistance in preparing data for review and analysis. In addition, the authors are very grateful to the Deanship of Scientific Research and Research Center, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.

Stratton TP, Palombi L, Blue H, Schneiderhan ME. Ethical dimensions of the prescription opioid abuse crisis. The Bulletin of the American Society of Hospital Pharmacists. 2018;75(15):1145-50.
Gounder C. Who is responsible for the pain-pill epidemic. The New Yorker. 2013.
Meier B. Pain killer: A" wonder" drug's trail of addiction and death: Rodale; 2003.
Grassley C, Baucus M. Baucus Grassley opioid investigation letter to American pain foundation 2012 [Available from: https://www.finance.senate.gov/download/baucus-grassley-opioid-investigation-letter-to-american-pain-foundation.
Ornstein C. American pain foundation shuts down as senators launch investigation of prescription narcotics 2012 [Available from: https://www.propublica.org/article/senate-panel-investigates-drug-company-ties-to-pain-groups.
Kounang N, Goldschmidt D. Senate report says patient advocacy groups get kickbacks from opioid manufacturers: CNN; 2018 [Available from: https://edition.cnn.com/2018/02/12/health/senate-report-opioid-manufacturers-donations/index.html.
Dart RC, Surratt HL, Cicero TJ, Parrino MW, Severtson SG, Bucher-Bartelson B, et al. Trends in opioid analgesic abuse and mortality in the United States. New England Journal of Medicine. 2015;372(3):241-8.
Saha TD, Kerridge BT, Goldstein RB, Chou SP, Zhang H, Jung J, et al. Nonmedical prescription opioid use and DSM-5 nonmedical prescription opioid use disorder in the United States. The Journal of clinical psychiatry. 2016;77(6):772.
Frenk SM, Porter KS, Paulozzi L. Prescription opioid analgesic use among adults: United States, 1999-2012: US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics; 2015.
Kolodny A, Courtwright DT, Hwang CS, Kreiner P, Eadie JL, Clark TW, et al. The prescription opioid and heroin crisis: a public health approach to an epidemic of addiction. Annual review of public health. 2015;36:559-74.
CDC. Understanding the Epidemic: CDC; 2017 [Available from: https://www.cdc.gov/drugoverdose/epidemic/index.html.
CDC. Increases in Drug and Opioid-Involved Overdose Deaths — United States, 2010–2015: CDC; 2017 [Available from: https://www.cdc.gov/mmwr/volumes/65/wr/mm655051e1.htm.
Bose J, Hedden SL, Lipari RN, Park-Lee E. Key Substance Use and Mental Health Indicators in the United States: Results from the 2015 National Survey on Drug Use and Health. SAMHSA; 2016.
Watson D, Duwve J, Greene M, Weathers T, Huynh P, Nannery R. THE CHANGING LANDSCAPE OF THE OPIOID EPIDEMIC IN MARION COUNTY AND EVIDENCE FOR ACTION. https://www.rmff.org/: INDIANA UNIVERSITY RICHARD M. FAIRBANKS SCHOOL OF PUBLIC HEALTH AT IUPUI; 2018.
Alzeer AH, Jones J, Bair MJ. Review of factors, methods, and outcome definition in designing opioid abuse predictive models. Pain Medicine. 2017;19(5):997-1009.
White AG, Birnbaum HG, Schiller M, Tang J, Katz NP. Analytic models to identify patients at risk for prescription opioid abuse. The American journal of managed care. 2009;15(12):897-906.
Rice JB, White AG, Birnbaum HG, Schiller M, Brown DA, Roland CL. A model to identify patients at risk for prescription opioid abuse, dependence, and misuse. Pain Medicine. 2012;13(9):1162-73.
Raja U, Mitchell T, Day T, Hardin JM. Text mining in healthcare. Applications and opportunities. J Healthc Inf Manag. 2008;22(3):52-6.
Kukafka R, Bales ME, Burkhardt A, Friedman C. Human and automated coding of rehabilitation discharge summaries according to the International Classification of Functioning, Disability, and Health. Journal of the American Medical Informatics Association. 2006;13(5):508-15.
Ross M, Wei W, Ohno-Machado L. “Big data” and the electronic health record. Yearbook of medical informatics. 2014;23(01):97-104.
Pereira L, Rijo R, Silva C, Agostinho M, editors. Using text mining to diagnose and classify epilepsy in children. e-Health Networking, Applications & Services (Healthcom), 2013 IEEE 15th International Conference on; 2013: IEEE.
Hylan TR, Von Korff M, Saunders K, Masters E, Palmer RE, Carrell D, et al. Automated prediction of risk for problem opioid use in a primary care setting. The Journal of Pain. 2015;16(4):380-7.
Carrell DS, Cronkite D, Palmer RE, Saunders K, Gross DE, Masters ET, et al. Using natural language processing to identify problem usage of prescription opioids. International journal of medical informatics. 2015;84(12):1057-64.
Palmer RE, Carrell DS, Cronkite D, Saunders K, Gross DE, Masters E, et al. The prevalence of problem opioid use in patients receiving chronic opioid therapy: computer-assisted review of electronic health record clinical notes. Pain. 2015;156(7):1208-14.
IUH. About our system 2017 [Available from: https://iuhealth.org/about-our-system.
Ghaffarinejad A, Kerdegary M. Relationship of opioid dependence and positive and negative symptoms in schizophrenic patients. Addiction & health. 2009;1(2):69.
Regenstrief_Institute. nDepth 2016 [Available from: https://www.regenstrief.org/resources/ndepth/.
Regenstrief_Institute. Regenstrief Data Core [Available from: https://www.indianactsi.org/translational-informatics/data-core/.
Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. Journal of biomedical informatics. 2009;42(5):839-51.
Cornell. 42 CFR Part 2 - CONFIDENTIALITY OF SUBSTANCE USE DISORDER PATIENT RECORDS 2017 [Available from: https://www.law.cornell.edu/cfr/text/42/part-2.
Boyd A, Van de Velde S, Pivette M, Ten Have M, Florescu S, O’Neill S, et al. Gender differences in psychotropic use across Europe: Results from a large cross-sectional, population-based study. European Psychiatry. 2015;30(6):778-88.
Simoni-Wastila L. The use of abusable prescription drugs: the role of gender. Journal of women's health & gender-based medicine. 2000;9(3):289-97.
Carrasco-Garrido P, De Andres AL, Barrera VH, Jiménez-Trujillo I, Jiménez-García R. National trends (2003–2009) and factors related to psychotropic medication use in community-dwelling elderly population. International psychogeriatrics. 2013;25(2):328-38.
KFF. Opioid Overdose Deaths by Gender 2018 [Available from: https://www.kff.org/other/state-indicator/opioid-overdose-deaths-by-gender/?dataView=2&currentTimeframe=0&sortModel=%7B%22colId%22:%22Location%22,%22sort%22:%22asc%22%7D.
Jiménez AM, Molina MIS-M, García-Palma MB. Gender bias in addictions and their treatment. An overview from the social perspective. Procedia-Social and Behavioral Sciences. 2014;132:92-9.
Hohmann AA. Gender bias in psychotropic drug prescribing in primary care. Medical care. 1989:478-90.
Canan C, Polinski JM, Alexander GC, Kowal MK, Brennan TA, Shrank WH. Automatable algorithms to identify nonmedical opioid use using electronic data: a systematic review. Journal of the American Medical Informatics Association. 2017;24(6):1204-10.

APPENDICES.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

A Comparison Of Text Mining Versus Diagnostic Codes To Identify Opioid Use Problem: A Retrospective Study

Status:

Version 1

Abstract

Figures

A. Background

B. Methods

C. Results

D. Discussion

E. Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1

A Comparison Of Text Mining Versus Diagnostic Codes To Identify Opioid Use Problem: A Retrospective Study​

Status:

Version 1

Abstract

Figures

A. Background

B. Methods

C. Results

D. Discussion

E. Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1

A Comparison Of Text Mining Versus Diagnostic Codes To Identify Opioid Use Problem: A Retrospective Study