Participant characteristics
Twenty-four of 33 invited individuals participated in the study. Participants included men (10) and women (14); policy and practice experts (n=11) and research experts (n=13). Workshop attendees held appointments across seven universities or research institutes internationally. Three individuals declined the invitation to participate as they were unable to attend the face-to-face workshop. Six consenting individuals were unable to attend this particular component of the workshop.
Key findings
Emerging themes were grouped into three categories according to the study aims: A) experiences with and limitations of adaptation frameworks; B) views on increasing framework utility and practicality in real world settings; and C) opportunities for future research to enhance impact of adaptation to improve impact of public health programs. These are described in detail below, with de-identified quotes from participants illustrating key arguments.
A). Experiences with and limitations of adaptation frameworks
Participants discussed their experiences in using existing adaptation frameworks and their views on the limitations and applicability of such frameworks in practice.
i. Available frameworks lacking in detail on ‘how to’ adapt in practice
Broadly, participants agreed that the framework elements as summarised in the scoping reviews were lacking in detail. Most of the 11 elements found in the frameworks were not considered to be described in ways which help practitioners implement them into practice. For instance, it was not well understood how to ‘decide what needs adaptation’, or how to ‘implement’ or ‘evaluate’, despite some attempts to explicitly provide guidance by the frameworks. While there was a recognition that such details may be implied, a more explicit description of the adaptation elements that were applicable to practitioners was suggested to be required if the frameworks were to guide practice.
A lot of things/steps are assumed, so maybe we need to be more explicit (public health/behavioural scientist)
Although it looks like these are new things, but they may be framed/titled something else and not necessarily missing... Make them explicit. Even if they are not needed [for the particular project], it is still good guidance… (public health/behavioural scientist)
Despite this, participants also suggested that an overly prescriptive framework was unlikely to be useful for practitioners. Some flexibility in how the frameworks are applied was needed, depending on available resource/infrastructure of the ‘end-user’ or organisation undertaking the adaptation.
Adaptation occurs mostly at a service level, where they pick and choose what they are implementing based on resources…. an overly technical document is unlikely to be useful (practitioner)
Frameworks [often] provide guidance on what is optimal, not necessarily what’s feasible or desirable in practice (public health/behavioural scientist)
….a trap looking at the very planned instead of what actually happens in the real world, which are the important things (practitioner)
The above quotes illustrate a tension between the recognised need to provide detailed guidance on ‘how-to’ in order to make the frameworks more useful, and the reality of often unplanned day-to-day adaptation decisions by practitioners. Participants recognised that while current framework guidance may be too broad, and hence not useful for those undertaking adaptations, producing an ‘overly technical document’ that requires highly technical skills and increased resources for the operationalisation of adaptation would likely to limit its feasibility to apply in practice. Further, participants noted that guidance needed to be flexible and practical and be considerate of real-world constraints. Consequently, it was noted that future guidance documents on adaptation would need to better consider contextual factors (e.g. level of expertise, role, skills of the individual undertaking adaptation) that impact on users’ decision-making and ability to undertake adaptation if such documents were to be useful for practitioners.
Delivery partners will change essential components if resources and capacity force them to (implementation scientists)
I don’t see the list as being relevant to a service delivery unit or to final level end user, for example schools…they would not use this. The utility of the framework hinges on [understanding] context (practitioner)
ii. Current adaptation guidance does not adequately cover implementation
A further theme of discussion related to the lack of explicit guidance on when and how to apply the elements of adaptation to EBI implementation strategies. To ensure EBIs are adequately implemented and embedded into real world practice, effective implementation strategies are needed. Current adaptation frameworks however provide limited guidance on selecting, delivering, measuring and understanding the impact of implementation strategies. Participants noted the ‘duality’ between the EBI itself (i.e. the public health intervention with established efficacy) and the implementation strategy (i.e. the strategy to support implementation of the EBI into practice), something which is not explicitly acknowledged in current frameworks. Participants recognised them as distinct components of the adaptation process, requiring consideration both individually and concurrently. Differences in ‘core components’ and mechanisms of both an EBI and its implementation strategy were discussed. Core components in the adaptation literature refer to components of the interventions that are necessary to lead to the intended positive outcomes (on an individual, provider, systems level) and in principle should not be adapted [3]. As such, determining core components of the EBI and implementation strategy via mechanism analysis was seen as a critical to informing the adaptation process:
There is always a duality with EBI and implementation… should do mechanisms analysis (separately) and look at different designs for how we can test these. (public health/behavioural scientist)
There is a distinction between essential components of implementation delivery approach and of EBI, different measure of each and different process and outcomes- all of these things need to be measured…For implementation there are different stakeholders, two very distinct development approaches and very distinct evaluation approaches. (public health/behavioural scientist)
The second quote reflects an opinion shared by many, that while there may be some similar elements to guide adaptation of interventions and of implementation strategies, the mechanisms, stakeholders involved, associated outcome measures and evaluation approaches will be different for these two distinct components. It was also noted that for EBIs/policies/guidelines with high fidelity requirements, selection and adaptation of implementation approaches may be particularly important.
iii. Structural limitations including wording and ordering of common elements
The structure of available adaptation frameworks [9] was also discussed. Participants noted there were some missing elements, such as ‘engage with stakeholders’ (to precede stakeholder consultation), or ‘decide whether to adapt’ (to precede a decision on what requires adaptation). Some elements were found to be duplicative and/or overlapping, for example engaging stakeholders and experts was thought of as one element instead of two, and training staff was considered not a separate element, but rather a part of the implementation process. Participants also noted that adaptation in practice was an iterative and cyclical process, rather than the linear model depicted by the current guidance. There was some acknowledgment, however, that for the sake of guidance, some simplification of thinking and presentation of adaptation elements conceptually, as linear, was needed (i.e. a list of ordered adaptation elements as opposed to circuitous process).
How do we turn it [into] a practical [resource] as we have got an academic model? There is a gap there…we need something practical (to support service decisions) (implementation scientist)
We want to get to the point where policy makers know what to do…Linear is useful and is simple (implementation scientist)
The need to simplify adaptation frameworks was discussed hand in hand with how to make any framework more accurately represent actual adaptation steps and processes in practice. Some suggested that this might entail working closely with those implementing the intervention to streamline the recommended elements and then to furnish it with real-life examples that are relevant to those who adapt on the ground.
Giving real-world examples about what steps would look like. (We need some) flexibility with which steps to use and how they have been applied, to help show people the relevance of the element and how to apply it (implementation scientist).
Yes, this is a critical point, [the guide] needs to be iterative with examples from different settings, so people can understand it (public health/behavioural scientist)
B) Views on increasing framework utility and practicality in real world settings
While participants noted that some improvements to include additional detail and refinement of the structure and wording of current recommendations, there was agreement that most of the existing elements [9] were important and relevant. The main suggestions on how to improve the utility of current adaptation frameworks were: i) better consider decisions on how to implement in the adaptation process; and ii) identify guiding principles of adaptation. This was incorporated into a list of overarching principles, which was suggested by participants as important to consider during the adaptation process. These findings are described below.
i.Consider decisions on how to implement in the adaptation process
Some participants proposed that instead of thinking of adapting intervention and adapting implementation strategies as two separate processes, it could be considered as ‘two cycles’.
[adapting intervention and adapting implementation] Shouldn’t be considered as stepwise, linear… and not two separate columns, but two integrative cycles (practitioner)
Participants also discussed when to consider implementation approaches as part of adapting EBIs.
Same process, such as elements for EBI and implementation [are used] at the start when comparing what is being done originally. This needs to be done again after adaptation of program… Revisit implementation after adaptation is complete. (public health/behavioural scientist)
The above quote reflects a view shared by some participants, that processes related to adapting the intervention and implementation strategy (where known) should be considered concurrently and then reviewed again when the EBI has been adapted.
ii.Identify guiding principles of adaptation rather than produce an overly prescriptive framework
There was a general consensus that there should be a focus on identifying a set of overarching principles of adaptation, which could then be used together with a more prescriptive framework.
We need to determine key principles first, rather than taking an encyclopaedic approach. (methodologist)
We need pragmatic, overarching instructions for system changes (first) and (then can) operationalise them to specific setting/system. (practitioner)
It was proposed that the adaptation process and steps could then be tailored in collaboration with practitioners and end-users involved in the adaptation process, to the specific context and setting within which a program is being adapted. By co-designing with practitioners and tailoring to their needs, and by feeding consumer views into the process of adaptation, the guiding adaptation framework is likely to be practical but not overly prescriptive.
Is it practitioners we really need to hear from? If this is the real audience, then in the first instance we should hear back from them (public health/behavioural scientist)
ii. Important overarching ‘principles that could be considered in future adaptation processes
Participants recommended six overarching principles as important when considering adaptation in the real world.
- Identify parameters/drivers of adaptation—perhaps as a first step, or as part of a needs assessment. Parameters relate to boundaries or the context in which adaptation may occur (e.g. who is making adaptation decisions, what skills are available, characteristics of the EBI/implementation strategies, evidence-base); drivers relate to factors which may determine the need for adaptation (i.e. reasons for adaptation including lack of fit with end-user/community needs, cost restrictions, scale of the intervention). This would allow practitioners to decide whether adaptation is needed, reasons for adapting and what needs to be adapted. The framework developed by Stirman and colleagues [5], which classifies the adaptations into various levels including ‘whom’ is making a modification, ‘what’ is being modified (i.e. content, context, or training and evaluation) and at ‘what level of delivery’ will the modifications be made to was suggested as particularly useful to guide these considerations.
Drivers of adaptation are really important as they guide how to approach (public health/behavioural scientist)
[let’s think about] entry strategy… why are we adapting? (practitioner)
Parameters determine level, approach is crucial for adaptation (public health/behavioural scientist)
Consider the work of Stirman…It addresses issues such as by whom, what level, context modification, what has been modified, why. The process of adapting EBI will then require less work and guidance (implementation scientist).
- Explicitly identify the implementers (i.e. who will implement the adapted EBI) and implementation strategies early on in the process of adaptation and consult in an ongoing way
Identification of implementer is a critical step as they can communicate to us what will work/ what won’t throughout the whole process (practitioner)
- Identify and engage with relevant stakeholders (e.g. providers, policy makers, practitioners, program implementers) early on, and throughout the adaptation process
(We need to)… engage appropriate stakeholders before consultation. After an appropriate level of engagement, this will help with consultation process. (public health/behavioural scientist)
- Commence adaptation by selecting an adaptation framework and/or using other available guidance/resources. Tailor application of framework to the end-user context and need and allow for flexibility in applying the framework/adaptation process. Terminate the adaptation process where necessary.
Allow flexibility in applying framework … need to consider process for getting out/stopping (public health/behavioural scientist)
Put in place termination process …. Like a stopping rule (practitioner)
- Consider costs of implementing the adapted strategy. This could include an explicit economic assessment/pre-modelling of costs/resources needed.
Need to understand the resources needed to implement the adapted EBI. First, undertake economic assessment…pre-modelling to see if it’s cost effective and what budget is needed, and then [understand] health service impact… can health service afford to implement the adapted version? Economic modelling guidelines for how to do it [understand costs] are not good or do not exist. (methodologist)
- Conduct both evaluation and ongoing monitoring of both the adapted EBI and implementation strategy.
Monitoring and sustainability currently missing from frameworks (methodologist)
C. Opportunities for future research to enhance impact of adaptation on public health programs
Participants also identified a number of opportunities to undertake research to advance the field of adaptation. Opportunities identified were:
Undertake empirical research to explore/examine the impact of adapting different components of an EBI/implementation strategy on efficacy and cost outcomes. This provides the opportunity to understand the usefulness of applying formal adaptation frameworks, recognising that many adaptations can be unintended. Further, there is a need to better understand the role of cost/formal economic analysis in supporting the adaptation process.
A lot of [adaptation] research is theoretical – we need to start establishing creative ways of building the empirical evidence. Possible to bed process within existing trials and do comparisons. (Implementation scientist)
[we need to] start understanding the cost and risk in finding a cheaper (adapted) way of delivering a service…for example a nurse instead of a practitioner. To what extent is improved cost (worth) risking the effectiveness? (Implementation scientist)
Cost, budget, financial theme not covered [in current frameworks] (practitioner)
…need to understand the resources needed, using economic assessment: pre-modelling to see if it’s cost effective and what budget is available; and understanding health service impact – can the service afford to implement?(methodologist)
Examine methods that identify core/non-core components of an intervention and its implementation strategy. Introduce formal mechanistic processes where possible (e.g. head to head trials, mechanistic evaluations, mixed methods).
Mechanism analysis is critical for adaptation of implementation strategies…Intensive and tailored strategies make it hard to have one size fits all, which is challenging for scaling. Mixed methods approach needed (public health/behavioural scientist)
Good implementation is evident within efficacious RCTs, so a mechanism analysis should be done [in situations of high implementation fidelity] to identify main contributors for effect of the EBI. This would show what the real core elements are, not the necessarily those hypothesised (practitioner)
Develop systems or data collection processes to capture and measure planned/unplanned adaptations/fidelity to the inte
rvention and implementation strategies. This can provide contextual information on how adaptation may affect the research translation process. The classification framework developed by Stirman et al was again suggested as a useful way to classify any modifications made to an EBI or implementation strategy.
A main challenge for our service is the ability to systematically capture data. (We are) currently running a large implementation trial [with processes] to capture adaptation using Stirman framework to work out where adaptation is occurring at each level (practitioner)