Implementation Science and Randomized Clinical Trial Conceptual Frameworks
This study will incorporate the effectiveness-implementation hybrid trial type 2 design.17 Under this framework, our study will have the coprimary aims of determining the effectiveness of the clinical intervention and establishing the intervention’s feasibility and potential utility.17
This study will also use the RE-AIM framework to conduct the evaluation. The RE-AIM Framework is well suited for technology innovation projects because it focuses on external validity in study design and guides the planning, conduct, evaluation, and maintenance of the intervention.18 In this project, the RE-AIM framework will be used to derive practical measures of how well the intervention works in real-world clinical settings and to produce a 360-degree assessment of its efficacy.
Overall Study Design
We will evaluate the effectiveness of the intervention in a pragmatic, cluster-randomized controlled trial, randomized at the provider level. The rationale for this is that the behavioral economic and usability components of intervention apply at the provider level; additionally, PCPs are expected to learn from the CDS, and thus there would be potential for contamination if randomization occurred at the patient level. The primary endpoint of the study is the change in mean SBP between baseline and 6 months compared across arms. Secondary endpoints include hypertension-specific process measures, process measures for CKD quality of care, adverse drug events, and hypotension. We also hope to assess whether the intervention improves process measures for quality of CKD care such as annual urine albumin tests. The overarching structure supporting the evaluation is the RE-AIM framework, and this protocol was reported in accordance with SPIRIT guidelines (Fig. 1).
Study Setting
The Brigham and Women's Primary Care Practice-Based Research Network (BWPC PBRN) is one of 155 PBRNs nationally certified by the Agency for Healthcare Research and Quality (AHRQ). The BWPC PBRN is a network of 15 practices which includes hospital-based practices, community-based practices, and community health centers affiliated with Brigham and Women’s Hospital.
Eligibility Criteria
Patients: All patients over the age of 18 who have a visit with a PCP at one of the intervention practices during the 2 years preceding the visit date will be eligible for enrollment. The first inclusion criteria will be CKD, defined as two prior eGFRs 16–59 mL/min/1.73m2 separated by 90 days, as calculated by CKD-EPI, or two prior UACR > 30mg/g. The second inclusion criteria will be uncontrolled hypertension, defined as at least one SBP > 140 mmHg within the 2 years preceding the enrollment visit, as well as SBP > 140 mmHg at the enrollment visit. Patients with a most recent eGFR ≤ 20 or two previous eGFRs within 2 years separated by at least 90 days ≤ 15 will be excluded.
Providers: Our objective is to include PCPs who have a consistent panel of primary care patients and represent primary care providers broadly outside of the academic medical center setting, and thus our criteria excludes providers who only see urgent care and walk-in patients. We also excluded residents in training. We compiled a list of currently employed physicians, physician assistants and nurse practitioners in our primary care network. The network includes 220 PCPs who care for approximately 150,000 patients. Of the 220 PCPs, 184 were eligible to be enrolled, and the remainder were excluded; no PCPs opted out of the trial.
Study Intervention
One component of the intervention is a series of Epic Best Practices Advisories (BPAs). These CDS are designed to utilize computable phenotypes (CPs), defined as disease definitions or algorithms that allow curation of disease populations using data from electronic health records (EHRs).19 We developed five CPs, each with its own CDS recommendation. The first CP includes patients with CKD and uncontrolled SBP for whom it is advised that an angiotensin converting enzyme inhibitor (ACEi) be prescribed. The second includes patients with CKD and uncontrolled SBP for whom an angiotensin receptor blocker (ARB) should be prescribed. The third CP includes patients who are currently on an ACEi but not at an optimal dose, while the fourth CP includes those who are on a suboptimal dose of an ARB. The fifth CP includes patients who are maximized on an ACEi or ARB, but are not on a diuretic. The CDS prompts PCPs to make clinical decisions specific to these phenotypes. For example, the PCP of a patient exhibiting the first CP would be prompted to prescribe a starting dose of an ACEi (e.g., lisinopril), while the PCP of a fourth CP patient would be prompted to increase the dose of the patient’s currently prescribed ARB (e.g., losartan).
Another aspect of the multicomponent intervention, developed utilizing user-centered design principles, is the display of patient specific data explaining why the CDS fired. Our team performed contextual inquiry sessions and two rounds of usability testing (group design and individual think aloud sessions) on the CDS prototype. The main focus of the contextual inquiry sessions was to assess the different activities, steps, and thinking processes involved in managing uncontrolled blood pressure using EHRs and any other resources used by PCPs during a patient encounter. The goal of the group design sessions was to validate the requirements gathered during the contextual inquiry sessions and gain a greater understanding of user preferences and mental models. The goal of the individual think aloud sessions was to uncover usability issues and validate design decisions. Additionally, hyperlinks to clinical guidelines supporting the CDS recommendation are included in the final CDS.20 21
Finally, a pre-commitment email was sent to all participating PCPs asking them to pledge to follow recommendations about blood pressure management. We will evaluate whether the pledge makes people more likely to either follow the recommendation or to add an accountable justification (stated rationale for why one disagrees with the recommendation) when interacting with the BPA, incorporating the behavioral economic principle of accountability into the intervention.
Implementation of CDS in Epic
Each of the rules were added to the Epic database. The BPAs appear at the time of chart opening by the PCP if criteria are met (see example in Fig. 2). The CDS was be moved to the Production environment in “silent mode” for approximately six weeks before the scheduled start date of the trial, where a report recorded when it would fire but it was not be displayed to the user. When the trial begins, we will identify control patients in real time according to the same inclusion criteria as intervention patients. This will allow us to validate that the rules are accurately identifying patients and producing the correct recommendations through a chart review. The CDS was also activated in the Production environment for a pilot study.
In practice, the CDS will fire only on the initial enrollment visit, except if the PCP chooses the “remind me next visit” option in the “acknowledge reason” section. However, if the patient’s SBP has dropped below 140/90 mmHg, the patient is already on the recommended medication or a new allergy to an ACE/ARB has been entered, this follow-up firing will be suppressed.
Outcomes and Measures
Within an implementation-effectiveness framework, a hybrid trial type 2, defined as the simultaneous testing of a clinical intervention and an implementation intervention/strategy, will be conducted.17 This decision is motivated by the intervention’s multiple components, ranging from aspects of clinical effectiveness (management of HTN in CKD patients) to usability principles and strategies (PCP adherence to CDS guidelines). Additionally, outcomes and measures will be guided by the RE-AIM framework.
Reach will refer to the overall use of the CKD CDS, including the number of PCPs and patients for whom it fires. To assess the reach of the CDS, statistics on the quantity and types of firings will be collected through enterprise data warehouse (EDW) and Epic queries. Concurrent manual review of Epic reports and chart review on CDS firing statistics will be conducted by team members to verify the automated monthly summaries. Analytic variables will include the percentage and types of clinicians in primary care who use the software, descriptions of excluded clinicians, PCP review and/or response to pledge email, PCP interaction with the CDS, signing of orders or accountable justification documentation within the CDS, and whether the BPA fired appropriately during encounter.
Effectiveness will refer to the clinical efficacy (process and outcome measures), usability of the software in the primary care environment, process measures, and both positive and negative unanticipated consequences. In evaluating effectiveness, the primary endpoint is the change in mean SBP between baseline and 6 months compared across arms. This outcome was chosen as the primary outcome because of the growing need in primary care to monitor patients’ HTN in order to help mitigate negative long-term outcomes of CKD such as kidney failure, cardiovascular events, and death. A meta-analysis of three large cohorts of CKD patients without diabetes concluded that maintaining blood pressure below 140/90 mmHg decreases risk of these outcomes significantly.22 Several guidelines have been issued to emphasize the importance of HTN control in CKD.23–25 Additional secondary outcomes are listed in Table 1 and Table 2.
Table 1
Outcome Variables and Measures for Both Arms
Measurement Variable | Form of Variable | Analysis Metric | Time Point from First Visit |
Primary |
Mean SBP | Continuous | Change from baseline | 6 months |
Secondary |
Mean SBP | Continuous | Change from baseline | 12 months, 18 months |
Controlled SBP Rate | Dichotomous | Proportion of patients with controlled SBP rate | 6 months, 12 months, 18 months |
Urine Albumin to Creatinine Ratio | Continuous | Urine Albumin to Creatinine Ratio | 6 months, 12 months, 18 months |
Serum Creatinine > 2.0 | Dichotomous | Proportion of patients with Creatinine > 2.0 | 6 months |
Smoking Status | Dichotomous | Proportion of patients who smoke | 6 months |
eGFR | Continuous | eGFR | 6 months, 12 months, 18 months |
Medication ordered | Dichotomous | Proportion of patients with recommended medication ordered | 6 months |
Basic metabolic panel ordered | Dichotomous | Proportion of patients with basic metabolic panel ordered | 6 months |
Referral to e-consults | Dichotomous | Proportion of patients with referral to e-consults | 6 months |
BPA acceptance | Dichotomous | Proportion of patients for whom BPA interaction was not an acknowledge reason | 6 months |
Mean SBP of less than 110 | Dichotomous | Proportion of patients with mean SBP of less than 110 | 6 months |
Newly documented allergy | Dichotomous | Proportion of patients with newly documented allergy due to adverse drug events | 6 months |
K + > 5.2 | Dichotomous | Proportion of patients with K + > 5.2 | 6 months |
K + < 3.6 | Dichotomous | Proportion of patients with K + < 3.6 | 6 months |
Mean SBP analysis with imputation of missing 6-month BP measurement | Continuous | Change from baseline | 6 months |
Controlled SBP Rate analysis with imputation of missing 6-month BP measurement | Dichotomous | Proportion of patients with missing 6-month BP measurement | 6 months |
Mean SBP as-treated analysis (patient level) | Continuous | Change from baseline in patients whose provider ordered the recommended medication | 6 months |
Mean SBP as-treated analysis (provider level) | Continuous | Change from baseline in patients whose provider ordered the recommended medication | 6 months |
Table 2
Outcome Variables and Measures for Intervention Arm Only
Measurement Variable | Form of Variable | Analysis Metric | Time Point from First Visit |
Acknowledge reason entered | Dichotomous | Proportion of patients with acknowledge reason entered | 6 months |
Feedback button clicked | Dichotomous | Proportion of patients with feedback button clicked | 6 months |
PCP participation on pledge email survey | Dichotomous | Proportion of patients whose PCPs participated in pledge email survey | 6 months |
Guideline accessed | Dichotomous | Proportion of patients with guideline accessed | 6 months |
Adoption will refer to the percentage and types of settings and staff that embrace the innovation. We will analyze the various ways in which the intervention PCPs accept, reject, and generally interact with the BPA. Using a BPA report extracted through Epic, we will collect which CDS fired, the date and time of the firing, the MRN and demographic information of the patient on whom it fired, and the user (PCP). We will also extract the user follow-up action (whether the PCP indicated they would order a medication or basic metabolic panel through the BPA) or which acknowledge reason the PCP chose. If “other” is indicated as an acknowledge reason, the PCP will be asked to elaborate with a comment. Finally, in conducting our manual chart review, we will record whether the BPA firing for the patient was appropriate and whether the course of action that the PCP indicated they would take through the BPA differed from the course that they actually took.
Implementation will refer to the consistency of CDS use, any support resources used, any barriers and/or enabling factors that are identified, any workarounds to barriers that develop, any changes from pre-intervention to intervention period, and any unintended consequences to patient safety or workflows. Prior to the clinical trial, a pilot study was conducted in live clinical settings. The BPAs were turned on for approximately two weeks for all intervention providers, and the patients for whom the BPAs fire during this period were classified as “Pilot Patients.” Interaction with the BPAs was monitored. The first time a BPA fired for a PCP, the research team contacted the PCP by email to gather feedback through a survey or an interview. The experiences of these PCPs were noted, but no changes were made to the BPA. However, some early workarounds that were discovered included selecting the accountable justification “Other” without a valid reason (e.g., “x”) in order to circumvent the BPA, as well as ordering a medication or panel without completing the order by signing off on it.
Maintenance will refer to how well the innovation components and their effects are sustained, as well as any strategies that are used to uphold the intervention over time. This will be recorded by qualitative descriptions of system performance longitudinally, emerging workflow changes, and long-term unintended consequences, and how the BPA fit into the existing PCP workflow. The methods used will include contextual inquiry sessions, interviews, and surveys.
Sample Size and Power Calculation
According to our analysis of patients who saw a PCP at one of the 15 BWPC practices in 2009,
there were 3,118 patients with two prior eGFR 16–59 ml/min/1.73m2 measured at least 90 days apart between 2007-2008.70 We determined that 42% (n = 1309) of these patients had at least two BP > 140/90 mmHg. Those patients with elevated SBP had a mean SBP of 153.9 mmHg and standard deviation of 14.0 mmHg. We will assume that 71% of the enrolled patients will have at least one follow-up visit during the 6 month follow-up period.26 According to an analysis of 2013 data, 56,461 patients visited a PCP at one of the 15 BWPC practices in 2013.27 We identified 5,593 patients with CKD who visited a PCP during 2013, when CKD was defined as two prior eGFR < 60 ml/min/1.73m2 or UACR ≥ 30 mg/g measured at least 90 days apart between 2007–2012. Combining our findings from the two studies, we expect 42% of these 5,593 patients to fit both of our inclusion criteria, so N = 2,349 patients. We based our power calculation on an expected decrease in the mean of the final SBPs for patients in the intervention arm of at least 3 mmHg as compared to the mean of the final SBPs in the control arm, which is a clinically important decrease. The two arms will be compared using a robust generalized estimating equations z-test for continuous data; this approach does not assume normality of the outcome, and accounts for a possible cluster effect of patients within PCP.28 Using the GEE z-test with a 2- sided type I error rate of 2.5%, we calculated that 497 evaluable patients per arm and an average of 6 patients per PCP would provide over 80% power to detect an average 3 mmHg SBP decrease in the intervention arm. We assumed an intra-cluster (clinician) correlation coefficient of 0.1, as is commonly assumed in this type of cluster randomization study.29 Therefore, we have power to detect a 3 mmHg decrease in mean of final SBPs in the intervention arm.
Recruitment
This is a pragmatic clinical trial. Once the study period begins, each patient who has an office visit with a PCP and fulfill criteria for CKD and uncontrolled HTN will be automatically enrolled in the study. Both inclusion criteria will be assessed, in real time, for every adult patient who visits a PCP. The patients will be designated as belonging to the intervention or control arm based upon the PCP that the patient sees. The subgroup of PCPs included in the pilot study will remain enrolled in the clinical trial; the patients they saw during this period will be excluded from the final analysis.
Allocation
This study will utilize a matched-pair cluster randomized design with the intervention on the cluster level, and the main outcome (6-month minus baseline change in SBP) measured at the patient level. We will have 174 clusters (made up of 184 clinicians) in the study (Fig. 3). Clusters were made up of either one PCP with their own panel of patients, or two PCPs who share a panel, also known as a “co-management dyad.” We will match pairs of clusters with similar number of patients and prior year mean blood pressure of patients in the cluster. One cluster in each pair with be randomized to the intervention and unit one to usual care.
Data Collection and Follow-Up
Patients will be electronically identified and included in the study over the course of 12 months. Patients seen by PCPs during the pilot study will be excluded. Retrospective data indicates that 70% of patients have a follow-up around 6 months. Outcomes assessment will occur at 180 days (+/- 60 days). After the 12-month enrollment period ends, data collection will continue for 6 months so that those enrolled toward the end of the enrollment period will have a full 6 months to complete any interventions ordered by the PCP. Clinical outcomes will be recorded and reviewed every month over the course of the trial.
Statistical Methods
Using the Haybittle–Peto approach, the conventional p-value of 0.05 can be used in the primary analysis (difference in final SBP between intervention and control arm PCP panels). In other words, the p-value for the GEE z-test must be ≤ 0.05 for the intervention and control groups to be declared significantly different. Descriptive statistics (e.g., means and standard deviations, medians and interquartile ranges, frequencies) will be used to examine the distributions of demographic (e.g., age, sex, race/ethnicity) and clinical characteristics (e.g., comorbidities, baseline eGFR, smoking status). We will compare intervention and control groups on baseline characteristics with Rao-Scott chi-square tests for categorical variables (accounting for clustering) and GEE z-tests for continuous variables (accounting for clustering).30 The primary endpoint is the change in mean SBP between baseline and 6 months compared across arms and the primary analysis will be an intent-to-intervene analysis. SBP at intermediate visits will not be used in this analysis. We expect the baseline SBP mean and standard deviation to be the same in both arms. If there is a difference at baseline, we will use a difference in differences approach.
Secondary Analyses
We will perform a secondary analysis for patients who did not have a subsequent visit after the enrollment visit. We will use multiple imputation to estimate the final SBP for patients who have missing data. We will perform another secondary analysis, as "as-treated" analysis, on patients where the provider chose to order the recommended medication. We will also perform a secondary analysis at the provider level. This "as-treated" analysis will include only providers who did not choose to opt out after randomization.
Ethical Considerations
The Human Subjects Institutional Review Board at Brigham and Women’s Hospital approved this study protocol, and a Data Safety Monitoring Board will ensure the ongoing safety of the trial. Potential harm outcomes will be monitored and addressed as needed. Any relevant protocol modifications will be to relevant parties.