The persistence of unacceptably low rates of translating research findings into practice has led to increasing attention to implementation research (IR) as a means to significantly accelerate improvements in public health.1,2P Over a decade ago, Eccles and Mittman (2006) defined IR as “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services and care.”3 Similarly, the National Institutes of Health has consistently defined IR as “the scientific study of the use of strategies to adopt and integrate evidence-based health interventions into clinical and community settings in order to improve patient outcomes and benefit population health.”4,5 Considering the significant amounts of funding, time, and effort invested in IR, it would be ideal if the field of IR had developed one or more scientific theories as defined by National Academy of Sciences (i.e., a comprehensive explanation of the relationship between variables that is supported by a vast body of evidence).6 Although the field of IR has developed various theories, models, and frameworks to support IR, these theories, models, and frameworks have some limitations.7,8 First, they tend to be narrow in scope, focusing on one area of implementation research (e.g., evaluation, implementation determinants, implementation processes), instead of comprehensive explanations of phenomena. Second, although there are many models and frameworks, there are few theories (comprehensive explanations of relationships between variables) and, to our knowledge, no IR theories are supported by vast bodies of evidence the way prominent theories in other fields are (e.g., Theory of Planned Behavior, which has been widely applied across fields to predict human social behavior and has a vast body of evidence, including meta-analyses, assessing the predictive validity of its theoretical propositions).9,10 Given the limitations of IR theories, efforts to accelerate the development of theories that meet National Academy of Sciences standards are warranted.
Research priority setting (i.e., promoting consensus about areas where research effort will have wide benefits to society) is one approach to accelerating research advancements.11 The Priority Aims and Testable Hypotheses for IR (PATH4IR) Project seeks to accelerate IR on several of IR’s priority domains, aims, and testable hypotheses via estimating the extent to which IR to date has examined these priority areas. Helping accelerate IR on these priorities should accelerate IR’s development of National Academy of Sciences-defined scientific theories, which in turn will help accelerate improvements in public health. Below we identify, describe, and justify four priority domains, three priority aims, and four priority testable hypotheses for IR, which are the focus of the PATH4IR Project and its scoping review.
Four priority domains for implementation research
A plethora of IR theories, models, and frameworks have identified numerous IR domains.7,12 Table 1 lists the domains of three IR models/frameworks that have guided much IR to date. Building on these IR models/frameworks,13-15 other IR,16 principles of data reduction,17 and a general framework for moderated mediation,18 the PATH4IR Project identified four priority domains for IR. Each priority domain is defined below and in Table 2.
Table 1. Domains included in several existing implementation research models/frameworks
Implementation Research Model/Framework
|
List of Domains
|
Proctor et al. (2009) – A Conceptual Model of Implementation Research15
|
Intervention Strategies; Implementation Strategies; Outcomes
|
Damschroder et al (2009) – The Consolidated Framework for Implementation Research14
|
Intervention Characteristics; Outer Setting; Inner Setting; Characteristics of the Individuals Involved; Process of Implementation
|
Aarons et al. (2011) – Conceptual Model of
Evidence-Based Practice Implementation
in Public Service Sectors13
|
Outer Context; Inner Context; Innovation Characteristics & Intervention Developers; Innovation/System Fit; Innovation/Organization Fit; Interconnections
|
Table 2. The priority domains for implementation research
Priority Domain (acronym)
|
Brief Description
|
Justification
|
Implementation Strategies (IS)
|
Strategies used to put into practice a program of known dimensions (e.g., EBP)
|
15,19
|
Evidence-Based Measures of Implementation (EBMI)
|
A measure shown to be predictive of improvement in one or more key HHROs (e.g., client outcomes)
|
21
|
Health and Health-Related Outcomes (HHRO)
|
End-points regarding evidence-based process of care, client/patient outcomes, or population outcomes
|
15, 22
|
Context-Related Moderators/Mediators (CRMM)
|
Measures of the outer setting/context or inner setting/context that are hypothesized to moderate and/or mediate relationships between the other domains (i.e., IS, HHRO, EBMI)
|
13, 14
|
Note: IS = Implementation Strategies; EBMI = Evidence-Based Measure of Implementation; HHRO = Health and Health-Related Outcomes; CRRM = Context-Related Moderators/Mediators; EBP = Evidence-Based Practice;
Implementation strategies. Implementation strategies are defined as the strategies used to put into practice a program of known dimensions (e.g., an evidence-based practice [EBP]).15,19 Given how IR has been defined and that implementation strategies are the quintessential independent variable in IR,3-5 we consider the implementation strategy (IS) domain a priority for IR.
Evidence-based measures of implementation. If implementation strategies are the quintessential independent variable of IR, implementation outcomes have become the quintessential dependent variable. However, consistent with the important distinction demonstrated between a practice and an EBP,20 an important distinction has been demonstrated between an implementation outcome and an evidence-based measure of implementation (EBMI).21 An implementation outcome is defined as “the effects of deliberate and purposive actions to implement new treatments, practices, and services,”16 whereas an EBMI is defined as “an implementation outcome measure that is predictive of improvements in key client outcomes” (i.e., health and health-related outcomes [HHROs], such as client functioning, health-related quality of life, or morbidity/mortality).21 This means that while all EBMIs are implementation outcomes, not all implementation outcomes are EBMIs. IR has historically prioritized implementation outcomes, but as noted by Proctor and colleagues (2009), implementation outcomes should not be treated as dependent variables until we have advanced them as consistent, valid, and efficient measures of implementation.16 Otherwise, we rely on the assumption that implementation outcomes are predictive of HHROs, without empirically demonstrating this to be true. To our knowledge, the PATH4IR Project is the first to explicitly identify EBMIs as a priority domain for IR.
Health & health-related outcomes. Health outcomes (e.g., client/patient functioning) and health-related outcomes (e.g., health-related quality of life, quality adjusted life years) are the outcomes that IR seeks to ultimately improve. Despite this, not all outcome-focused IR models/frameworks explicitly include the HHRO domain.13,14 Instead, many focus on implementation outcomes, leaving out HHROs entirely. We identify HHROs as a priority domain for IR for two reasons. First, as noted above, until EBMIs are established, measuring only implementation outcomes relies on the assumption that implementation outcomes are predictive of HHROs. Second, as noted by Foy et al. (2015), “If studies evaluating the effects of implementation interventions are to be of relevance to policy and practice, they should have end-points related to evidence-based processes of care.”22
Context-related moderators/mediators. Moderation occurs when the effect of an independent measure on a dependent variable depends on the level of another measure and mediation occurs when the effect of an independent variable on a dependent measure is transmitted through a third variable.23 Given that existing IR models/frameworks have highlighted the importance of context13,14,24 and that Edwards and Lambert’s (2007) general framework for moderated mediation18 guided identification of the priority domains for this project, context-related moderators/ mediators (CRMMs) was identified as a priority domain for IR. Conceptualizing context as potential moderators/mediators (instead of just discrete factors that “influence” implementation) moves the field of IR towards National Academy of Sciences-consistent theory as it starts to clarify relationships between constructs.
Three priority aims for implementation research
There are numerous aims (i.e., research questions) that IR could address, and there is value in establishing consensus regarding the types of aims IR should prioritize. Relative to IR’s domains, IR’s aims have received less explicit attention. The work of Curran et al. (2012)25 is one exception. Specifically, for their type 3 effectiveness-implementation research categorization, Curran et al. recommended that the primary aim of this research category was to “determine utility of an implementation intervention/strategy” and the secondary aim was to “assess clinical outcomes” ). associated with implementation trial.”25 Curran et al. also recommended implementation outcomes (e.g., adoption, fidelity) as dependent measures for the primary aim, with client outcomes (e.g., patient symptoms patient functioning) as dependent measures for the secondary aim.25 However, priority aims have not generally been explicitly addressed by most other IR models/frameworks.13-15 Given that developing or contributing to generalizable knowledge is central to how research is defined,26 it is important that IR prioritize aims that seek to develop or contribute to generalizable knowledge for its priority relationships. Thus, building from the four priority domains described above, we identified the following three priority aims for IR: (1) the IS to HHRO relationship (i.e., IS à HHRO), (2) the IS to EBMI relationship (i.e., IS à EBMI), and (3) the EBMI to HHRO (i.e., EBMI à HHRO). Consistent with the mediational analysis literature,27-30 we have termed IR focused on the IS à HHRO relationship as Path C IR (the red triangle of Figure 1), IR focused on the IS à EBMI relationship as Path A IR (the blue triangle of Figure 1), and IR focused on the EBMI à HHRO relationship as Path B IR (the green triangle of Figure 1). Each priority aim is defined below and in Table 3.
Table 3. The priority aims for implementation research
Priority Aim
|
Type
|
Advance generalizable knowledge regarding the
IS à HHRO relationship
|
Path C
implementation research
|
Advance generalizable knowledge regarding the
IS à EBMI relationship
|
Path A
implementation research
|
Advance generalizable knowledge regarding the
EBMI à HHRO relationship
|
Path B
implementation research
|
Note: IS = Implementation Strategies; HHRO = Health and Health-Related Outcomes;
EBMI = Evidence-Based Measures of Implementation.
Advance generalizable knowledge regarding the IS à HHRO relationship. Advancing generalizable knowledge about the relationship between an IS and a HHRO is termed Path C IR. Given IR’s emphasis on strategies to increase the uptake of EBPs to improve patient and population health3-5 and the importance of measuring outcomes that have relevance to policy and practice,22 Path C IR was identified as a priority aim for IR. An example of Path C IR is a 29-site cluster randomized implementation experiment Garner et al. (2012) conducted between 2008 and 2012 that focused on testing the impact of a pay-for-performance implementation strategy to improve the implementation and effectiveness of the Adolescent Community Reinforcement Approach (A-CRA), which is an EBP for adolescents with substance use disorders.31 The dependent variable of interest was for the a primary HHRO, which was adolescent substance use recovery status at 6-month follow-up.
Advance generalizable knowledge regarding the IS à EBMI relationship. Advancing generalizable knowledge about the relationship between an IS and an EBMI is termed Path A IR. Given that an EBMI is a measure of EBP implementation found to be predictive of a key client outcomes21 Path A IR was identified as a priority aim for IR. Relative to IR that has tested the impact of an IS on implementation outcomes that do not have evidence of being a meaningful predictor of key client outcomes, IR testing the impact of an IS on EBMIs appears be limited. Having established an EBMI for A-CRA as part of an effectiveness study,32,33 Garner et al. (2012)31 also provide an example of Path A IR. Indeed, examining the impact of pay-for-performance on an EBMI called Target A-CRA (i.e., 10+ of the core the A-CRA components delivered within no less than seven sessions), which prior research found to be significantly associated with greater reductions in adolescents’ days of abstinence at follow-up,32 Garner et al. (2012) found that relative to adolescents in the implementation-as-usual condition, adolescents in the pay-for-performance condition had a significantly higher likelihood of receiving Target A-CRA.31
Advance generalizable knowledge regarding the EBMI à HHRO relationship. Advancing generalizable knowledge about the relationship between an EBMI and HHRO is termed Path B IR. Research by Nosek et al. (2015),34 which increased concern regarding the reproducibility of psychological science, underscores why Path B IR is a priority. That is, it is important that significant relationships (e.g., EBMI à HHRO) supported as part of effectiveness research be examined for replicability within IR. As part of their IR experiment, Garner et al. (2012)31 provide an example of Path B IR by replicating a significant association between Target A-CRA (i.e., the previously established evidence-based measure of implementation) and adolescent abstinence from substance use at follow-up (i.e., the HHRO).31
Four priority testable hypotheses for implementation research
While the possible testable hypotheses for IR are numerous, there is value in establishing consensus regarding the types of testable hypotheses IR should prioritize. Toward helping generate National Academy of Sciences-defined scientific IR, prioritizing one or more of the four testable hypotheses shown in Figure 2 is warranted. More specifically, there is a need to prioritize IR testable hypotheses regarding the extent to which an IS has demonstrated one or more of the following, relative to an appropriate active-control implementation strategy: superior effectiveness (upper left quadrant [ULQ]) and/or cost-effectiveness (upper right quadrant [URQ]), non-inferior effectiveness (lower left quadrant [LLQ]) and/or cost-effectiveness (lower right quadrant [LRQ]). Each priority testable hypothesis is described below and in Table 4.
Table 4. The priority testable hypotheses for implementation research
Priority Testable Hypothesis
|
Type
|
Cost-effectiveness hypotheses from a superiority trial
|
URQ hypotheses
|
Effectiveness hypotheses from a superiority trial
|
ULQ hypotheses
|
Effectiveness hypotheses from a non-inferiority trial
|
LLQ hypotheses
|
Cost-effectiveness hypotheses from A non-inferiority trial
|
LRQ hypotheses
|
Note: URQ = Upper Right Quadrant; ULQ = Upper Left Quadrant; LLQ = Lower Left Quadrant;
LRQ = Lower Right Quadrant.
Effectiveness hypotheses from a superiority trial. Testing the extent to which an experimental IS has superior effectiveness, relative to an active-control IS, is termed IR testing an upper left quadrant (ULQ) hypothesis. In contrast to research on clinical treatments, where an active-control condition may not exist or be appropriate, IR should include the most appropriate active-control IS possible. One of the most appropriate active-control condition IS may be the IS used as part of an EBPs effectiveness research. To date, the “large and growing evidence base relating to the effectiveness of implementation strategies” noted by Foy et al.22 has tested ULQ hypotheses and supports that this testable hypothesis is and should remain a priority for IR. Indeed, given that tests of ULQ hypotheses may continue to be the most common type of IR hypotheses, it may not be much longer before results of ULQ hypothesis tests are analyzed as part of a meta-analysis.
Cost-effectiveness hypotheses from a superiority trial. Testing the cost-effectiveness of an IS that has been shown to have superior effectiveness, relative to an active-control IS, is termed IR testing an upper right quadrant (URQ) hypothesis. It is considered a priority testable hypothesis for IR as knowing the effectiveness of an intervention/strategy is not sufficient for many potential users, especially decision makers who need to know whether the benefits from the intervention/strategy are commensurate with its costs (i.e., whether it delivers value),35-38 Further, noting that economic evaluation of implementation strategies “has been neglected,” Foy et al. encouraged IR with an economic evaluation component.22 Building upon Garner et al. (2012),31 which found pay-for-performance to be an effective IS for improving the implementation and effectiveness of A-CRA in a superiority trial, Garner et al. (2018)39 provide an example of IR testing an URQ hypothesis. Supporting the cost-effectiveness of a pay-for-performance IS, Garner et al (2018)39 found that although the pay-for-performance strategy led to 5% higher average total costs compared to the implementation-as-usual control condition, this average cost increase of 5% resulted in a 325% increase in the average number of patients who received Target A-CRA (i.e., the EBMI).39
Effectiveness hypotheses from non-inferiority trial. Testing the extent to which an experimental IS has non-inferior effectiveness, relative to an active-control IS, is termed IR testing a lower left quadrant (LLQ) hypothesis. Similar to how Schumi and Wittes (2011)40 explain non-inferiority, testing a non-inferiority hypothesis seeks to provide evidence that the IS being tested is “not unacceptably worse” than the IS being used as a control. This is a priority for IR given strategies used to study a clinical intervention’s effectiveness may not be possible in practice settings (e.g., too intensive). We are not aware of IR that has tested LLQ hypotheses. However, a close example is a non-randomized observational IR study by Stirman et al. (2017)41 that compared two strategies for providing post-workshop consultation in an evidence-based cognitive therapy. As detailed by Stirman et al., results of their study did not support the hypothesis of the group consultation and feedback condition being non-inferior to the gold-standard individual feedback condition.41
Cost-effectiveness hypotheses from non-inferiority trial. Testing the cost-effectiveness of an IS shown to have non-inferior effectiveness, relative to an active-control IS, is termed IR testing a lower right quadrant (LLQ) hypothesis. Again, given decision makers desire to know the extent to which benefits from an IS are commensurate with its costs,35 LLQ hypotheses were identified as a priority for IR. Although not from the field of IR, an example of testing cost-effectiveness hypotheses from a non-inferiority trial is provided by Bansback et al. (2018),42 which extended research by Oviedo-Jockes et al. (2016)43 to support the non-inferiority of injectable hydromorphone hydrochloride (i.e., a narcotic pain reliever) relative to injectable diacetylmorphine hydrochloride (i.e., pharmaceutical heroin).
Objectives
The primary objective of the PATH4IR Project’s scoping review is to advance understanding regarding the extent to which IR to date has examined the four priority domains, three priority aims, and four priority testable hypotheses described above. We hypothesize that IR addressing these priorities will be limited (i.e., represent significant gaps in the extant IR literature). Thus, a secondary objective of this review is to help advance understanding regarding what domains, aims, and testable hypotheses IR has focused on to date.