This study was conducted as a secondary analysis of administrative data using a net benefit approach.
Data Source
Using data files obtained from the Georgia Department of Public Health for the years 1999 through 2012, we constructed a retrospective cohort by deterministically linking hospital discharge data, supplied to the Georgia Department of Public Health from the delivery hospitals, for all singleton delivery hospitalizations to birth, fetal death, and maternal death certificates using a unique maternal identifier embedded in the files by the Georgia Department of Public Health Office of Health Indicators for Planning. The linking methodology has been previously described.10 All data was obtained with the permission of the Georgia Department of Public Health and the procedures were approved by the Emory University Institutional Review Board.
High Risk Identification Models
Two models for identifying women at high maternal risk were used to create two unique samples.
The experimental sample was created using the weighting of 20 comorbid conditions included in the OCI.6 The index provides a score by summing individual weights for each condition, the weights having been derived from the beta coefficient in the model. The score has been validated to improve the prediction of maternal end organ damage compared to the Charlson Comorbidity Index, and the OCI has been validated with hospital discharge data in a separate sample.6,11 Scores for the OCI in these data ranged from 0-12 with a mean score of 0.55 (SD 0.90). This was different from the score range from 0-19 and mean score of 0.91 (SD 1.42) in the validation cohort when the OCI was created.6
To use the OCI as a sample selection tool, a cut-off value to indicate high maternal risk was selected by finding the highest net benefit using a score of 2 or higher. As this was the first test of the method, the cut-off value was selected using the population available in these data. While this method prevents generalizability of the cut-off value, it was considered appropriate because the goal of this study was to test the usefulness of the method for sample selection, not validate the cut-off value. The cut-off value to indicate high maternal risk status was selected using net benefit analysis as defined for the main analysis. The cut-off with the highest net benefit was a score of four (Net Benefit of 6 per 100,000); cut-off values less than four had negative net benefits while cut-off values greater than four became progressively closer to zero (See Table 1).
Table 1: Results of Net-Benefit Analysis to select a cut-off value for the Obstetric Comorbidity Index
Cut-Off Value
|
Sensitivity
|
Specificity
|
Positive Predictive Value
|
Net Benefit per 1,000 live births
|
2
|
0.52
|
0.87
|
0.02
|
-185.7
|
3
|
0.38
|
0.96
|
0.04
|
-0.17
|
4
|
0.16
|
0.99
|
0.06
|
0.63
|
5
|
0.16
|
0.99
|
0.08
|
0.33
|
6
|
0.03
|
0.99
|
0.09
|
0.12
|
7
|
0.01
|
0.99
|
0.12
|
0.05
|
The comparison group was created using dichotomous identification of any comorbid condition included in the OCI. Dichotomous identification of any comorbidity on a list is the method currently used to stratify fetal and maternal risk in the literature, though the specific list of conditions varies between studies.12,13 By applying the conventional practice with the same conditions used for the OCI, this study compared the value of identifying risk with the index summary score rather than comparing the comorbid conditions.\
Comorbid conditions were identified using The International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic codes in the hospital discharge record, which is consistent with prior literature addressing high maternal risk. The authors of the OCI provided the full list of included ICD-9-CM codes in their publication.6 These codes were used without alteration for the experimental group and the control group.
Predicted Outcome
The predicted outcome for this study was poor maternal outcome which was defined as either severe maternal morbidity or maternal mortality. These data allowed identification of severe maternal morbidity during delivery hospitalization using the hospital discharge record, while maternal mortality was identified using the death certificate and included deaths up to 42 days postpartum.
Maternal mortality is the death of a woman during pregnancy or the postpartum period. For this study, maternal mortality was limited to direct obstetric deaths as defined by the World Health Organization and identified by International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10) codes on the maternal death certificate.14 The use of direct obstetric death allows a reproducible measure of maternal mortality beyond delivery hospitalization and is limited to deaths related to pregnancy.
Severe maternal morbidity was calculated using a standard algorithm that identifies maternal end organ damage from ICD-9-CM diagnosis and procedure codes.15 This algorithm updated previous lists of codes that identified specific complications and used length of stay less than the 90th percentile to eliminate diagnosis codes that may have been used to “rule out” conditions. When compared to the gold standard of medical record review, this method had a sensitivity of 77% for identifying severe maternal morbidity.16 The most common problem with this algorithm is that the ICD-9-CM code for transfusion has a high rate of false positive because it is unable to discriminate between the presence of any transfusion and the presence of a transfusion of four units. This difference is important because transfusion of at least four units indicates severe maternal morbidity in the algorithm. To prevent overestimation of severe maternal morbidity, this study did not include the ICD-9-CM code for transfusion in the severe maternal morbidity algorithm. A sensitivity analysis was performed that included the ICD-9-CM code for transfusion to identify the potential extent of underestimation due to this change in calculation.
Analysis
The samples created by each method were described by the number of women identified as being at high maternal risk, along with the method’s sensitivity, specificity, positive predictive value, accuracy, and odds ratio for a poor maternal outcome.
The samples were compared for their ability to create a useable research sample of women whose physical condition would warrant transfer to a higher level of maternal care. The comparison was performed with net benefit because, unlike assessment of accuracy or area under the curve, net benefit analysis does not assume the benefits and risks of misclassification are equal.9 Net benefit was calculated using the formula
Net Benefit = True Positives/n - False Positives/ n (pt/1-pt)
where n is the total population from which the sample is being selected and pt is the probability of being identified as high risk. In net benefit analysis, the method with the highest net benefit is considered “superior.” A model in which no woman is identified as high maternal risk is represented by a net benefit of zero; so any model with a negative net benefit indicates that model performs worse than identifying no woman at high risk.17