An Idea Worth Sustaining: Evaluation of the sustainability potential of Choosing Wisely across Ontario Community Hospitals and Family Health Teams

doi:10.21203/rs.3.rs-63468/v1

Download PDF

Research

An Idea Worth Sustaining: Evaluation of the sustainability potential of Choosing Wisely across Ontario Community Hospitals and Family Health Teams

https://doi.org/10.21203/rs.3.rs-63468/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Evidence shows that 30% of healthcare services are unnecessary and unlikely to benefit patients and may even be harmful. Choosing Wisely (CW) is a physician-initiated, international campaign (with campaigns in at least 20 countries) aimed at addressing unnecessary medical care through the provision of evidence-based resources and clinical guidance. Our objectives were to evaluate the sustainability potential of CW across four Ontario community hospitals and affiliated family health teams (FHT).

Methods

We recruited CW implementation teams across Ontario hospitals and their affiliated primary care family health teams (FHTs) to participate in a mixed-methods study to identify and respond to their CW implementation and sustainability challenges. The study involved the administration of a validated sustainability Survey at baseline, 6- and 12-months follow-up, and participation in two focus groups (at baseline and 6 months) to discuss identified successes and opportunities for improvement, and to respond to implementation and sustainability challenges through a team exercise involving the co-creation and implementation of an action plan to directly address them.

Results

Fifteen CW implementation teams across five Ontario community hospitals and affiliated FHTs participated in the study. Three CW de-adoption priority areas were investigated by teams: de-prescribing of proton pump inhibitors (PPIs), and reducing Pre-Op testing and BUN/Urea lab testing. All 15 teams achieved a baseline sustainability team score that was well above the threshold of what is considered a potentially sustainable innovation (i.e., >55%). We observed steady improvements in sustainability scores over three time points across all primary care teams, which is a strong indication that CW is sustainable in these settings. Regardless of site or priority area, facilitators that were common across all teams were fit with existing processes and workflows, leadership support, and optimized team communication. Common challenges were: lack of awareness and buy-in, lack of leadership engagement or a champion, and lack of fit with existing workflow and culture. All teams identified at least one challenge (during the baseline focus groups) for which they co-designed and implemented a plan to address them aimed at maximizing the sustainability potential of their CW priority areas.

Conclusions

Health Economics & Outcomes Research

Sustainability

Implementation

Mixed-methods

Achieving sustainability is an iterative, dynamic process that can best be achieved over time by continuously identifying and responding to implementation challenges until the new way of doing things becomes usual practice or part of routine care
The NHS Sustainability Survey yields valuable information when used alongside team discussions to optimize implementation efforts but shouldn’t be used in isolation
Communication and awareness of an innovation such as CW is key to sustainability because people need to feel that what they are doing is making a difference and that the additional effort is worthwhile.
Choosing Wisely (CW) teams across Ontario community hospitals were empowered to identify and respond to implementation challenges over time to maximize the sustainability potential of their CW initiative

It is estimated that 30% of health care services, tests, treatments and procedures are unnecessary and unlikely to benefit patients^1–3, and can also lead to harm for patients, wasted resources, and longer wait times for care^3–7. Choosing Wisely (CW) is a physician-initiated, national campaign aimed at reducing overuse by addressing low value care that is unlikely to benefit patients or even cause harm, focusing on reducing overuse of care^8,9. The goal of the campaign is to facilitate the communication between physicians and patients about unnecessary tests, treatments and procedures through the provision of evidence-based resources and clinical guidance⁸, and to help them to make informed decisions to ensure high-quality care⁹. The campaign has now spread internationally to at least 20 countries¹⁰ including the development of recommendations to reduce unnecessary tests, treatment and procedures¹¹ .

The successful implementation and spread of evidence-based de-implementation innovations such as CW and their potential to improve care can only be achieved if they are sustainable^12–13. This is particularly important for CW because despite the overwhelming worldwide commitment to its adoption, its effect on decreasing unnecessary care has been slow^14–16. While this is not surprising to some extent given the wide range of CW priority areas, settings and dissemination and implementation efforts, “de-implementation” of care services (particularly when part of usual care or routine use) is not well understood but has been defined as the “abandonment of medical practices or interventions that have been found to be ineffective and harmful”¹⁷. We need to identify the factors that influence behaviour change in the context of de-implementation. Sustainability is a key implementation outcome and has been defined as: “the degree to which an innovation continues to be used after initial efforts to secure adoption are completed”¹⁸ or becomes a routine part of practice to support continuous care delivery^12,19−20. Ensuring sustainability capacity is also needed to scale up innovations²¹, which can increase the impact of effective innovations through “vertical diffusion”, a deliberate, systematic approach to increasing coverage, range and sustainability^22,23.

Although sustainability is an important consideration during the implementation of innovations including those aimed at de-implementation such as CW, it is seldom considered or not considered early enough in the implementation process^24,25. It is not surprising that up to 40% of all new programs or interventions don’t last beyond the first 1–2 years after initial funding^26,27. Another challenge is that investigation of the long-term sustainability of innovations is lacking^{13,20,24,28,29}, focusing mainly on short-term outputs, neglecting the long-term, decision-informing outcomes needed for practice and policy domains^25,30. These deficiencies can lead to implementation failure and wasted resources^{4–6,31−33}, and worsen patient outcomes and quality of care^6,24,34,35. Increasing “awareness” about low value care is not enough to achieve sustainable health impacts. We need to demonstrate long-term benefits of effectiveness and uptake of innovations through behaviour and practice change, demonstrate increased patient knowledge and acceptance of these new innovations, and to show value to the system through decreased health care utilization^9,10. The success of ongoing implementation of innovations is therefore largely dependent on their sustainability potential. Critical to this assessment are to understand the determinants of sustainability, to involve and engage all relevant knowledge users who are part of the process to implement the innovation, and to empower the implementation team to address identified barriers. This type of assessment has the best potential to facilitate implementation efforts of change initiatives such as CW and to support sustained improvements in health care services and patient outcomes over time¹³. The objectives of our study were to: (i) assess the sustainability potential of CW across Ontario community hospitals and their affiliated primary care sites; (ii) to work with CW implementation teams to optimize their efforts; and (iii) to evaluate our overall process to advance the science of sustainability.

Design overview

Our objectives were addressed using a 12-month, multicenter, sequential mixed-methods design³⁶ (Fig. 1) involving three iterative phases of investigation across three time points: T1 (baseline); T2 (3-month follow-up) and T3 (final follow-up). Research and ethics board (REB) approval was attained through Clinical Trials Ontario (CTO) for NYGH, Markham Stouffville Hospital (MSH), St. Joseph’s Health Centre (SJHC) and Southlake Regional Health Centre (SLRHC). REB approval for Michael Garron Hospital (MGH) was attained through their REB office. The reporting of our study was guided by the Standards for Reporting Implementation Studies (StaRI) checklist³⁷.

Recruitment of CW teams

In Ontario, North York General Hospital (NYGH) was an early adopter of CW and began implementing the initiative in 2014. In 2016, the hospital received funding to spread CW to a network of Ontario community hospitals (Joint Centres for Transformative Health Care Innovation), which is a partnership between NYGH and six large community hospitals and 13 affiliated primary care centres in Ontario. The goal of the Joint Centres is to improve the quality and value in health care, and to share innovative practices. Leveraging the governance structure of the Joint Centres advisory board consisting of hospital and primary care clinic and administrative leads, we used a purposive sampling strategy to identify teams that were involved in the implementation of a CW initiative (e.g., de-prescribing proton pump inhibitors [PPIs], reducing BUN/Urea lab testing) at their sites between 2017–2018. Team members could be front-line healthcare providers (i.e., nurses, pharmacists, physicians, trainees), decision-makers (i.e., chiefs, directors, managers) and researchers. We identified 24 potentially eligible teams from five community hospitals (NYGH, MSH, SJHC, MGH SLRHC) and six FHTs affiliated with these hospitals. We held information sessions with each team to explain the study process including participants’ roles and expectations, to identify additional members of their CW implementation team, and to enrol them into the study.

PHASE 1: Baseline Assessment of sustainability (T1)

We used a two-step process to assess the baseline sustainability potential of each team’s implementation of their CW initiative.

Step 1: NHS Sustainability Survey and Model: Sustainability survey: With the help of site administrative and clinic leads, we emailed a validated sustainability survey developed by the UK National Health Service (NHS) Institute for Innovation and Improvement Program³⁷ to all consenting CW implementation teams (n = 15). The NHS Sustainability survey is based on a validated model representing 10 factors across three domains that are considered important in sustaining change in healthcare³⁸. Respondents are asked to provide their perceptions on each of the 10 sustainability factors by indicating the level that best fit with their current implementation situation. The goals of the model and survey are to help teams self-assess their implementation efforts and to become aware of the challenges (and successes) associated with a new practice chance such as CW. More specifically, the survey helps reveal the strengths and weaknesses of implementation, and to predict the likelihood of sustainability. Teams were given two weeks to complete the survey with follow-up email reminders to non-responders at 2 and 4 weeks after the due date³⁹. Individual survey scores were combined to derive an overall mean team score. Analysis: To identify the best potential for improvement across the 10 sustainability factors, we calculated the largest difference between actual scores and maximum potential score (a pre-specified maximum score built into the NHS survey)³⁸; to identify which factors contributed to successful implementation, we calculated the least difference between actual score and maximum potential score. Individual survey scores are collated, and aggregate team scores can be generated. An overall sustainability score of > 55% is considered a “reason to be optimistic” (i.e., above the threshold of what is considered a potentially sustainable intervention); and a score of < 45 suggests that some action needs to be taken to “increase the likelihood that the improvement initiative will sustain”³⁸. We also investigated which sustainability factors were associated with the greatest potential for improvement (i.e., largest difference between team score and maximum potential sustainability score). Outcomes: Overall baseline mean team sustainability scores and mean scores for each of the 10 sustainability factors across the three domains of the NHS Sustainability survey.

Step 2. Action planning focus groups: Action planning focus groups: Two weeks after completing the NHS Sustainability survey, teams were invited to participate in one-hour action planning focus groups to obtain a more in-depth understanding of their CW implementation. Each team was given a summary report outlining the team’s overall baseline sustainability survey results (including scores and accompanying bar graphs) aimed at facilitating a discussion among team members on how to address identified challenges, and to select at least one challenge for which they could co-design an action plan that could be feasibly implemented within 3 months. The team survey scores were meant to start the “conversation” around identified sustainability challenges³⁸. Focus group questions were driven by the NHS Sustainability survey scores as well as questions embedded within the NHS Sustainability model, which offers practical advice on achieving sustained use of the initiative³⁸. The guide is meant to raise early awareness of sustainability challenges, and the opportunity for teams to iteratively address these challenges to optimize the new initiative’s potential for impact³⁸ (Outcome set 1). Another goal of the focus groups was to encourage teams to co-design a plan that would directly respond to identified challenges revealed by the survey (Outcome set 2). The in-person focus groups were facilitated by an experienced moderator who helped to formulate action-based recommendations and strategies to improve the sustainability potential of their respective implementation efforts, focusing on the areas of opportunity as identified through the survey analysis. The process was aimed at raising early awareness of sustainability challenges and the opportunity for teams to iteratively address these challenges to optimize the new initiative’s potential for impact. Analysis: Sessions were audio recorded and transcribed verbatim. Two reviewer pairs used a multi-step thematic analysis process⁴⁰ to independently read transcripts, create an initial list of codes, meeting to develop a codebook through consensus-based discussions, assigning codes to transcripts, discussing the main themes of the data with all team members, and deciding on how best to organize and represent data.

PHASE 2: Three-month follow-up (T2)

Three months after the baseline administration of the NHS sustainability survey and focus groups, we re-administered the survey to all teams to assess whether the action plans designed by teams were implemented and if any new challenges have emerged that needed to be addressed. We used the same methods and procedures for assessment as at baseline. We invited teams for a second focus group, to discuss the results of the survey and how it compared with baseline data (Outcome set 1). The action plan items were discussed in the context of how they were applied in the previous three months, and if any further changes needed to be made to address new challenges (Outcome set 2). At the conclusion of these focus group each team was given a summary report outlining: (i) any changes in survey scores from baseline and accompanying bar graphs; and (ii) a description of actions that were implemented in response to identified challenges from the baseline assessment.

PHASE 3: Final follow-up (T3)

Three months after the second administration of the NHS sustainability survey, we administered it at the conclusion of the study to identify any changes in sustainability scores over time. In addition, we included an evaluation form to assess the impact and value of our study and overall process to advance the science of sustainability (Appendix X). Survey questions: Survey questions consisted of 5-point Likert type questions (where 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree) and open-ended questions. Outcomes: Participants were asked to rate their agreement with statements about the CW sustainability study and its specific features and to provide any additional comments about the value and usefulness of the: (i) information sessions; (ii) the NHS Sustainability survey including its usability; (iii) focus group sessions (to understand the survey results, to identify the team’s sustainability challenges, and if they helped to formulate an action plan to address challenges); (iv) action plan report (readability and organization, whether the team used the information in the report to make changes to their CW implementation process); and (v) to provide an overall assessment of the feasibility of participating in the study (i.e., completion of three surveys and participation in two focus groups), satisfaction; and whether the information they received throughout the study has increased their knowledge about the concept of sustainability. Analysis: Descriptive statistics for quantitative data (frequencies, means) and content analysis⁴¹ for qualitative data. Qualitative data was also triangulated with quantitative assessments to support data interpretations. We used a meta-matrix method^42,43 to triangulate survey and focus group data to facilitate our interpretation of findings. We looked for commonalities and contradictions of themes emerging from the data, noting reflections, isolating patterns and processes, and gradually building generalizations⁴¹. Survey scores across the three time points were used to create line graphs to show changes in sustainability scores across time. We also sent a report for each team to facilitate each team’s implementation efforts outlining their setting profile, NHS sustainability survey results over time, top successes and challenges, focus group results (action planning to address challenges), and overall key points and lessons learned.

Site and participant characteristics

We conducted 12 information sessions with 15 potentially eligible teams across four hospitals and six FHTs. Of these, all 15 teams representing 91 individuals consented to participate. The site and participant characteristics of hospital and primary care teams are shown in Tables 1 and 2, respectively. A total of 173 surveys were completed across three time points: baseline (n = 67), first follow-up (n = 59) and final follow-up (n = 47). Twenty-nine focus groups were held in total; 15 at first follow-up and 14 at final follow-up. Teams consisted of different combinations of health care professionals, chiefs/directors/managers, and clinic and administrative staff: clinicians (nurses, physicians, pharmacists), trainees (residents, pharmacy students) and health information data managers.

Table 1

Participant and site characteristics of nine teams across four Ontario community hospitals that participated in the sustainable study
Site and team		Study time period (number of participants)			Implementation details		Team member roles at T1 (number of participants)
Site and team		T1 (n = 41)	T2 (n = 35)	T3 (n = 25)	CW priority area	Hospital Dept.	Team member roles at T1 (number of participants)
Hospital 1	Team 1	7	4	4	Pre-Op	Surgery	Clinical chief (n = 1) Anaesthesiologist (n = 1) Nurse (n = 2) CTM: Perioperative (n = 1) CTM: Surgical program (n = 1) Surgery program director (n = 1)
	Team 3	5	5	3	BUN/Urea	Laboratory medicine	Lab medicine manager (n = 1) Clinical informaticist; Nurse (n = 1) Chief of medicine; Medical director of critical care (n = 1) Chief of ED (n = 1) Director Lab Medicine (n = 1)
	Team 5	8	7	5	PPI	Pharmacy	Physician (n = 2) Physician; MRP (n = 1) Pharmacy director (n = 1) Pharmacist practitioner (n = 2) Pharmacist (n = 2)
Hospital 2	Team 8	3	4	3	BUN/Urea	Laboratory medicine	Nurse; Clinical lead (n = 1) Nurse (n = 1) Laboratory manager (n = 1)
Hospital 2	Team 9	5	5	3	PPI	Pharmacy	Pharmacist (n = 2) Student Pharmacist (n = 1) Physician (n = 1) Nurse; Professional Practice Leader (n = 1)
Hospital 3	Team 11	4	4	2	PPI	Pharmacy	Nurse practitioner (n = 1) Pharmacist (n = 1) Pharmacy student (n = 1) Physician (n = 1)
Hospital 3	Team 13	4	3	2	Pre-Op	Surgery	Nurse (n = 1) Anaesthesiologist (n = 1) Clinical program director (n = 1) Decision support consultant (n = 1)
Hospital 4	Team 14	3	3	3	BUN/Urea	Laboratory medicine	Clinical chemist (n = 1) Clinical Informatics manager (n = 1) Laboratory medicine director (n = 1)
Hospital 4	Team 15	2	DNC	DNC	IP Echo	Cardiac laboratory	Clinical informatics manager (n = 1) Cardiologist; Head of non-invasive cardiac lab and echo lab (n = 1)

Table 2

Participant and site characteristics of six teams across six family health teams (FHTs) affiliated with community hospitals that participated in the sustainable study
Site and team		Study time period (number of participants)			Team member roles at T1 (number of participants)
Site and team		T1 (n = 26)	T2 (n = 24)	T3 (n = 22)	Team member roles at T1 (number of participants)
FHT 1	Team 2	4	4	3	Administrative (n = 1) HI manager (n = 1) Pharmacist (n = 1) Physician (n = 1)
FHT 2	Team 4	4	2	2	Pharmacist (n = 1) Physician (n = 1) Dietitian; QI coordinator (n = 1) QI decision support specialist (n = 1)
FHT 3	Team 6	5	5	3	HI manager (n = 1) Pharmacist (n = 1) Physician (n = 2) Resident (n = 1)
FHT 3	Team 7	2	1	2	Nurse; HI manager (n = 1) Physician (n = 1)
FHT 4	Team 10	6	7	6	HI manager; QI administrator (n = 1) Manager (n = 1) Physician (n = 4) Pharmacist (n = 1)
FHT 5	Team 12	5	5	6	Patient care manager (n = 1) Physician (n = 1) Resident (n = 2) EMR system administrator (n = 1)

NHS sustainability survey

Table 3 shows the mean NHS Sustainability survey team scores over time (T1, T2, T3) across hospital and primary care sites and their respective CW priority areas (Figs. 2–5). The mean baseline team scores across all sites and teams ranged 65–87%, which is above the threshold of what is considered a potentially sustainable innovation (i.e., ≥ 55%)³⁸. Survey results are organized according to the CW priority areas investigated by hospital and primary care family health teams (FHTs):

Table 3

Mean NHS Sustainability survey scores (range) across primary care and hospital sites and Choosing Wisely priority areas from baseline (T1) to 6- (T2) and 12-months (T3) follow-up
Setting (number of sites) Total number of teams	CW priority area	Mean NHS Sustainability score (range); Number of participants				Change in scores over time
Setting (number of sites) Total number of teams	CW priority area	T1 (Baseline)	T2 (1st follow-up; ~6 months)		T3 (final follow-up; 12 months)	Percent change from baseline to T3
FHT (n = 6)	PPI	68% (53–74%) n = 26	73% (66–80%) n = 24		78% (70–83%) n = 22	10% increase
Hospital (n = 3)	PPI	65% (59–72%) n = 17	67% (64–69%) n = 16		67% (65–69%) n = 10	2% increase
Hospital (n = 3)	BUN/Urea lab testing	70% (61–89%) n = 11	72% (58–85%) n = 12		88% (72–91%) n = 9	18% increase
Hospital (n = 2)	Pre-Op testing	87% (82–92%) n = 11	83% (68–97%) n = 7		77% (57–97%) n = 6	10% decrease
Hospital (n = 1)	IP Echo testing	79% (73–85%) n = 2	Did not complete			Not applicable
Total number of participants		n = 67	n = 59	n = 47		Not applicable
NHS = National Health Service; FHT = family health team; PPI = proton pump inhibitor

Proton Pump Inhibitor (PPI) de-prescribing: FHTs: Six FHTs completed the NHS sustainability survey at baseline (n = 26), 3-month follow-up (n = 24) and final follow-up (n = 22). The baseline mean NHS sustainability scores ranged 53–74% and increased to a range of 66–80% at 6-months and to 70–83% at 12-months (Fig. 4). The mean sustainability scores increased from baseline to 12-month follow-up for all six teams (mean increase 10%; range 5–17%). Hospital teams: Three hospital teams completed the NHS sustainability survey at baseline (n = 17), 3-month follow-up (n = 16) and final follow-up (n = 10). The baseline mean NHS sustainability scores ranged 59–72% and increased at first follow-up to 64–69% (two teams) while the third team showed a 5% decrease from 72–67%. At 12-months follow-up, the mean sustainability scores overall increased from baseline for two teams (mean increase 6%; range 2–10%); and decreased for the third team by 4% (Fig. 3). This decrease may in part be explained by a few challenges that we identified from triangulating survey with focus group data: 1) incomplete and wide variability in participation by team members across the three time points due to staff and student turnover; 2) the program lead went on maternity leave (T2) and therefore was not able to complete the final survey; 3) the team has not implemented CW hospital wide. At the time of T1, the team started a pilot of the initiative only on their alternative level of care (ALC) floor. The team acknowledged that their current processes needed to be changed to ensure successful implementation and long-term sustainability once the initiative is rolled out hospital wide.

BUN/Urea lab testing: Hospital teams: Three hospital teams that implemented the BUN/Urea CW priority completed the NHS sustainability survey at baseline (n = 11), first follow-up (n = 12) and final follow-up (n = 9). Their baseline mean NHS sustainability scores ranged 61–89% and increased at first follow-up for two teams (range 72–85%) while decreasing by 3% for the third team from 61–58%. At 12-months of follow-up, the mean sustainability scores increased from baseline for all three teams (mean increase 13%; range 2–27%) (Fig. 4).

Pre-Op testing: Hospital teams: Two hospital teams that implemented the Pre-Op testing priority completed the NHS sustainability survey at T1 (n = 11), T2 (n = 12) and T3 (n = 9). Their baseline mean NHS sustainability scores ranged 82–92%. One of the teams showed a steady increase in sustainability scores over time with a 5% increase from T1 to T3 (Fig. 5) while the second team showed a 25% decrease in scores from T1 (82%) to T2 (68%) to T3 (57%). This large decrease may be attributed to several important challenges that we identified by triangulating survey with focus group data: 1) the implementation of their CW initiative did not occur until T2 (first follow-up); 2) implementation was extremely complicated at this site because they have a total of 55 surgeon’s offices not all of which are located at the hospital, and the hospital was in the midst of a merger with two other hospitals; 3) the site is not fully electronic/automated; 4) the Pre-Op patient care manager went on maternity leave midway through the study (T2), which affected the participation of the team, since this person acted as the point person to send out surveys and coordinate focus groups; and 5) the team’s high baseline sustainability team score (82%) may be explained by the team’s “hopefulness” for the initiative, since they have not yet implemented it at the start of the sustainability study (T1). A clinician member of the team indicated that the much lower scores observed in the later parts of the study may have been due to the burden of the CW initiative not being felt by the team until they launched it (T2), the high learning curve of the initiative, more work for front line staff, and not having a fully electronic/automated system to accommodate CW processes.

IP Echo: Hospital team: One hospital team implemented the IP Echo CW priority. The team had two members who completed the NHS sustainability survey at baseline (mean sustainability score range 73–85%) and first focus group but did not complete T2 and T3 surveys or final focus group.

Top challenges and success factors

Figure 6 shows that the largest proportion of implementation teams perceived fit with existing processes and workflows as the success factor for sustainability regardless of the CW priority areas that we investigated (i.e., Hospital/FHT PPI de-prescribing, hospital BUN/Urea testing and Pre-Op testing) followed by leadership support and optimized team communication. In terms of challenges, the largest proportion of teams indicated (regardless of CW priority area) that lack of awareness and buy-in, lack of leadership engagement or a champion and lack of fit with existing workflow and culture (Fig. 7). There was a 78% overlap in challenges between FHT and hospital teams that implemented the PPI de-prescribing priority. Among the six FHTs, challenges identified by the majority (50% or more) of teams were lack of leadership engagement or a champion, frequent staff turnover, volume of QI initiatives, and the time consuming and resource intensive nature of PPI de-prescribing. Among hospital PPI teams, two thirds found lack of awareness and buy-in and training as top challenges.

Qualitative Focus Groups

Analysis of 29 focus groups revealed three themes related to sustainability facilitators (Appendix A) and four themes related to challenges (Appendix B). All teams identified at least one challenge for which they co-designed and implemented an action plan to maximize the sustainability potential of their CW priority area.

Sustainability facilitators

Fit with existing processes and workflows

Four primary care teams indicated that CW is an intuitive campaign because it’s built into the system so even if teams change, the project can continue. Both primary care and hospital teams indicated that PPI de-prescribing is well supported by CW because it includes tools that support optimized implementation (e.g., the ability to flag which patients should be reviewed for PPI de-prescribing, algorithms and tapering schedules, and electronic order sets that include a default order for a pharmacist’s consult for a full medication review). For Pre-Op testing, hospital teams indicated that CW is a straightforward, standardized approach, and can increase the efficiency of clinical workflows (i.e., less paperwork for administrative staff and more flexibility in nurse scheduling). BUN/Urea teams perceived ease of implementation due to its ability to be implemented almost anywhere, and the relative straightforwardness of the process (i.e., since the test can be removed from the post-surgery order set, it can eliminate “automatic” or “default” ordering).

Leadership support

Three primary care teams identified leadership support for successful PPI de-prescribing likely attributed to supportive clinical and senior leaders and staff involvement and that the initiative was prioritized by clinical leadership as a patient safety issue. Since pharmacists are central to the PPI de-prescribing process, this also freed up the involvement of family physicians, although having clinical champions was perceived as important for success. For the two hospital PPI teams, support from senior leadership on an organizational level was perceived to be the reason for success. Two Pre-Op testing hospital teams indicated that change could not have been possible without clinical lead champions and Joint Centres support, and a BUN/Urea team attributed success to corporate level leadership.

Optimized team communication

One FHT has taken opportunities to present data on their de-prescribing experience, which fostered an environment to encourage staff involvement. A hospital PPI team experienced enhanced team collaboration and communication because of regular communications between pharmacists and providers to discuss PPI de-prescribing and re-assessment of therapy were facilitated by the CW initiative. They also found that available CW publications helped support changes in the process (de-prescribing algorithm, pamphlets for patients and pharmacists). Two BUN/Urea teams created CW posters, which has been effective in creating/raising awareness among ICU staff, which in turn has facilitated reduced ordering of these tests.

Sustainability challenges and actions to address them

Lack of awareness

Many primary care and hospital PPI teams indicated that the CW initiative, its existing tools to facilitate implementation, its progress, results and successes are not being shared among staff. A hospital Pre-Op testing team indicated that not all patient information makes it into charts and it’s not always clear why patients are receiving certain tests. A BUN/Urea test team was concerned that communications about the CW initiative may not have reached all affected stakeholders. Actions taken by teams to address these challenges included sharing information by the senior leadership using their newsletters and meetings (FHTs); creating a simple, one-page infographic of their clinic’s PPI results (FHT); incorporating PPI de-prescription as part of resident training (FHT); sending emails from the director to all stakeholders (BUN/Urea testing); holding educational events and meetings with staff prior to hospital-wide rollout (Pre-Op testing); making posters and handouts available for patients; and presenting final pilot study results to the pharmacy team as well as via publications, presentations and posters (Hospital PPI).

Lack of buy-in and engagement

Some primary care and hospital PPI teams questioned why more physicians are not engaged or why they are hesitant to engage in the CW initiative. To address this challenge, one primary care PPI team sent a mass email about PPIs which led to patient inquiries about de-prescribing and promoted uptake of PPI de-prescribing. Another uptake strategy was to encourage staff who regularly de-prescribe to meet with those who do not. A hospital PPI team had their senior administrative leadership present to the clinical leadership at operations committees and grand rounds to encourage buy-in. In the context of Pre-Op testing, lack of buy-in was perceived as staff (who had been there longer) being inadvertently more resistant to implementing the change; this was attributed to the time needed for behaviour change to happen. Lack of buy-in was identified as a challenge by all three hospital BUN/Urea teams. One team thought that there is lack of clear evidence underpinning patient-level benefits for removing this lab test, which made it more difficult to “sell” the initiative. To overcome this, some teams will perform an interrupted time series analysis to evaluate the impact of their implementation. Another team felt that poor uptake stemmed from nursing staff not understanding the reason for reducing BUN/Urea testing because they are not adequately engaged in the initiative. To overcome this challenge, the team engaged in conversations with and listened to frontline staff about the importance of the initiative, engaged in education and presentations to gain support and consensus. Another BUN/Urea team found it difficult to change engrained behaviour patterns and habits of staff, who may just “automatically select blood work on the standard order sets”. To acknowledge this challenge, the team conducted an “audit and feedback” exercise to review blood work volumes of various clinics and departments and shared these results. The team’s clinical lead has reached out to physicians to engage in a conversation about the appropriateness of BUN ordering, and the team’s lab manager also reached out to clinic teams to support behaviour change efforts.

Lack of leadership engagement or a champion

Three primary care PPI teams indicated challenges related to leadership engagement. One team had no Chief to enforce the initiative, and therefore had no consequences or levers when physicians did not want to participate. To overcome this challenge the team suggested trying to rotate clinical leadership. However, this was difficult because of the time-consuming nature of getting new leaders up to speed on the initiative, and different leaders may have variable passion or investment in the project. Both primary care and hospital PPI teams acknowledged the importance of having a physician champion or a clinical leader to motivate other physicians to engage in the initiative, as well as more accountability in senior leadership on the “front lines” to push the initiative. A hospital Pre-Op team also expressed the need to engage senior leaders in the initiative. A BUN/Urea test found it difficult to get clinician leaders from other units engaged. To overcome this challenge the team’s clinical lead reached out to the department clinical leaders across the hospital sites via email and conversations, and their quality improvement specialist gathered data to help support these engagement efforts. Another BUN/Urea team found that despite their efforts to intentionally involve and engage clinical leadership, it was difficult to obtain feedback from them because a champion was lacking for this initiative. To address this challenge, the team set up an ongoing dialogue (via their Medical Advisory Council) to engage site leadership prior to the final changes in the order sets, and to consider any concerns, reservations and order set exclusions. Once the final changes were complete, the administrative director sent communications to all medical directors, managers and clinical resource leaders.

Lack of fit with existing workflows and culture

One primary care team felt that the PPI de-prescribing initiative is not yet part of the organizational work culture, and they would like it to become a habit just like other process such as checking blood pressure. To do this, they are carrying out a number of interventions including education, communication, monitoring with audits and feedback. The FHT is also continuing to support a research project (which could be done by residents) to look at aggregate PPI data at different sites to provide more evidence for physicians on the benefits of the initiative. They also noted that non-academic primary care sites will struggle to incorporate CW into their work culture because they don’t have the academic work culture, which tends to promote patient safety, include a pharmacist and data manager, and have a built-in system for staff training and improving processes – these are features that make FHTs amenable to initiatives such as CW.

Evaluation of our process

45 of the 47 individuals across 15 teams who participated in PHASE 3 of our study (T3) completed the evaluation survey (response rate 96%). Appendix C shows the dispersion of mean scores on the evaluation survey of participant perceptions of our study overall, the information sessions, the NHS sustainability survey, the action planning focus groups, and the action plan reports. Overall perception of the Sustainability study: Respondents agreed that their participation in the sustainability was feasible, helped strengthen the implementation of their CW priority area, increased their knowledge about the concept of sustainability, and were overall satisfied with the study (mean score range 4.0-4.3 out of 5). Although teams agreed that they would need to continue meetings to sustain or maintain their current activities (mean 4.0), it was perceived less likely that their teams would meet regularly about the CW priority (mean 3.9). Another FHT member described the implementation of an innovation such as Choosing Wisely as challenging in a “climate of multiple competing demands” that requires a “true collaborative approach to make successful”. Information sessions: Teams agreed that the information sessions were effective for introducing the CW sustainability sub-study, helped them to understand the objectives of the study and the NHS sustainability survey, and found the information session overall useful (mean score range 4.1–4.3). NHS Sustainability Survey: Respondents perceived the NHS Sustainability survey as easy to complete and understand, took a reasonable amount of time, and helped to learn about the sustainability study (mean score range 4.0-4.3). A small sub-set of respondents (7%) indicated that the NHS Sustainability Survey was “inflexible”, and not reflective of their actual circumstances, which made it difficult to answer the questions. Action planning Focus Groups: Respondents agreed that the action planning focus groups were overall helpful, well organized with effective facilitators and helped team members understand the survey results and to identify sustainability challenges (mean score range 4.0-4.3). There was slightly less agreement by respondents about the focus groups helping teams to formulate an action plan to address challenges (mean score 3.9). Team Reports: Action plan Reports summarized each team’s sustainability scores, top challenges and successes, and provided an operationalizable summary of the plan and processes co-created by the team to address challenges. Respondents agreed that these reports were easy to read and well organized (mean score 4.1). However, they had less agreement for using these reports to make changes to their CW implementation process, whether teams perceived that these changes led to improvements in their implementation; and whether three months was a long enough period to implement their co-designed action plans (mean score range 3.5–3.6).

Overall, our study showed that all 15 implementation teams across five hospitals and six affiliated primary care sites achieved a baseline sustainability team score that was well above the threshold of what is considered a potentially sustainable innovation (i.e., ≥55%)³⁸. We observed steady improvements in sustainability scores over three time points across all teams, which is a strong indicator that CW is sustainable in hospital and primary care settings for the CW priority areas that were investigated (PPI de-prescribing, BUN/Urea testing and Pre-Op testing).

Action planning focus group sessions revealed that regardless of site or CW priority area, facilitators that were common across all teams were Fit with existing processes and workflows, Leadership support, and Optimized team communication (Fig. 8); common challenges were Lack of awareness and buy-in, Lack of leadership engagement or a champion, and Lack of fit with existing workflow and culture (Fig. 9). All teams identified at least one challenge for which they co-designed and implemented an action plan to maximize the sustainability potential of their CW priority area.

Actions taken by teams to address Lack of awareness and buy-in included sharing information via newsletters, meetings and emails; creating a simple, one-page infographic; incorporating PPI de-prescription as part of resident training; holding educational events and meetings with staff; making posters and handouts available for patients; and to engage in publications, presentations and posters. To increase buy-in for patients, one FHT sent a mass email about PPIs which led to patient visits about de-prescribing. To encourage buy-in for staff, one clinic encouraged those who regularly engage in CW activities to meet with those who do not; another team’s clinic lead presented at operations committees and grand rounds; a team undertook a trial to evaluate the impact of their implementation; while others engaged in conversations with frontline staff, provided education and presentations to gain support and consensus; performed an “audit and feedback” exercise, and reached out to clinicians about the appropriateness of lab tests and to support behaviour change efforts. Actions taken by teams to address Lack of leadership engagement or a champion included trying to rotate clinical leadership; contacting and learning from sites which had experience and success with the initiative’s implementation; reaching out via email and conversations; engaging the QI specialist to gather data to help support engagement efforts; and to set up an ongoing dialogue to engage site leadership prior to the final changes in order sets. Actions taken by teams to address Lack of fit with existing workflow and culture included making the initiative become a habit (similar to taking blood pressure) by carrying out a number of interventions including education, communication, and monitoring with audits and feedback; supporting research projects, which could be done by residents; and to think about how to provide CW to non-academic sites.

The sustainability scores at baseline were about 5–22% lower for teams that implemented the PPI de-prescribing initiative (regardless of setting) compared with other CW initiatives that were implemented in hospital (BUN/Urea, Pre-Op, IP Echo). Our focus group data supports this observation as the majority of challenges (apart from the three that were common to all CW priorities) were clustered around the FHT and Hospital PPI priority areas (78% overlap). Figure 7 highlights this clustering effect, which may in part be explained by the level of complexity required to implement PPI de-prescribing compared with other CW initiatives that may be less complicated (e.g., removing lab or imaging tests from order sets). PPI de-prescribing requires behaviour change for implementation (both from the provider and patient perspective). For example, the prescription PPIs is in part driven by patients, who may not understand the benefits of stopping or demand a re-prescription from their provider if their gastrointestinal (GI) symptoms return. Patient perceptions can in turn influence provider prescription behaviours. From the provider’s perspective, complexity might stem from feeling pressured by patients to re-prescribe PPIs if their GI symptoms return, not being aware of the evidence (or lack of buy-in) for the de-prescribing effort, and not seeing the immediate benefits of stopping PPIs. These complexities highlight that we need to better understand not only the mechanisms of the implementation process (i.e., PPI de-prescribing), but also the various knowledge user perceptions, and to design interventions that best match or mitigate identified challenges. It also highlights the importance of engaging patients to contribute to the optimization of implementation efforts.

Implementation of complex interventions such as CW is important to achieve meaningful health impacts, but this is challenging. Several focus group participants indicated that implementation of CW and required behaviour changes takes time, particularly at the early stages (e.g., to assess what needs to change to support the new initiative, to set up systems). One team took action to anticipate implementation problems by taking extra time to figure out the processes prior to implementation and felt that it was important to do this to ensure long-term sustainability of their initiative. We also observed that a single change during implementation can have both intentional and unintentional consequences. For example, in the PPI de-prescribing initiative, family physicians recognized that patients have a much larger influence on prescribing decisions than expected. Patients are driving prescription decisions either by pressuring family physicians to be put back on PPIs or by “going around the system” to obtain a prescription through a GI specialist. Unintentional consequences are inevitable when implementing complex interventions, but this underscores the importance of understanding the determinants of usual practice and how the new way of doing things (i.e., CW) will influence improvement efforts. Another suggestion was to “not reinvent the wheel” by learning from others who have already implemented CW so that teams don’t have to start from scratch. Suggestions were to develop an information linking mechanism or platform to share experiences such as through an online portal, and to have knowledge brokers or implementation facilitators to support implementation efforts. Communication was identified by several participants as a “key to making it [CW] work”, which is in line with one of the common challenges that we identified across all sites and CW priority areas: “lack of awareness”. In particular, participants suggested that the modes and purpose of communication are varied but important to create awareness (of the CW initiative and its impact), buy-in, and to have information delivered by a credible source (champions) in a balanced way (right amount, right time), and to “keep it going until it becomes routine or established practice behaviour”. Given that “people are busy so it’s easy to forget”, participants also indicated that communication needs to be “creative” to be effective. For example, to come up with a simple and easy way to draw attention to the CW initiative for staff, as did a BUN/Urea team that placed an infographic on top of their nursing workstation.

We found that several aspects of our study were essential for measuring and encouraging sustainability that can be considered by others adopting an innovation such as the CW campaign: Information sessions. We held information sessions with all participating teams where we explained the study, highlighting the importance of sustainability and outlining expectations for study participation. These sessions helped solidify teams (not all team members had previously worked together) and taught them how to complete the NHS Sustainability Survey. Action planning. The NHS Sustainability Model does not include a process for creating dialogue among implementation team members, although this is encouraged. In response, we held formal focus groups with teams. These helped focus discussions (which may not otherwise have happed on their own) and led to the majority of teams to formulate an action plan in response to at least one sustainability challenge. Repeat measurements of sustainability. Bringing teams together in focus groups and asking teams to repeat the sustainability survey over 3 time points not only provided an opportunity to identify and address sustainability barriers but also became a reinforcing activity for teams to identify and respond to their implementation challenges. Using experienced facilitators. Experienced facilitators helped participants understand the underlying concepts of sustainability, the NHS sustainability model and the survey results, stimulated discussion, helped teams identify the challenges that could feasibly be addressed, and shared examples and experiences of other teams with similar processes or challenges. Generalizability of our findings: As part of our strategy for sustainability, we embedded a step to facilitate the maintenance of each team’s CW initiatives. Each team member received a team report along with a conceptual Implementation and Sustainability Guide (Appendix D) to help further support their current and/or new CW implementation efforts. Our future work will involve validating this guide so it can be applied by other implementation teams and sites.

Strengths and Limitations

Our study had many strengths. First, to our knowledge, we are the first to evaluate the sustainability potential of an implemented Choosing Wisely campaign. Most CW evaluations are focused on measuring the impact of the campaign and whether their implementation has been associated with reductions in low value care^9,44,45, while others have assessed awareness and perceptions of the campaign by providers⁴⁵. We contributed to the current limited knowledge of ensuring the sustainability of innovations by evaluating the sustainability potential of the CW campaign across five Ontario hospitals and FHTs. Sustainability is seldom considered in implementation efforts^24,25 or considered early enough to be able to address its potential challenges and barriers^25,30. In fact, there is a significant evidence gap in investigating the sustainability of effective interventions²⁴, and only a few examples illustrate the application of the NHS sustainability model/framework^20,21. Our work contributes to advancing the limited knowledge of sustainability and its methods and how implementation teams can be empowered to influence their implementation efforts. Although the NHS Sustainability Model includes probing questions to encourage the dialogue among implementation team members, there is no guidance on how to achieve this. We built on this work by creating a strategy to ensure full engagement of teams through action planning focus groups, using discussion questions informed by the sustainability surveys. Our involvement (i.e., mobilizing teams and providing opportunities for a solutions-focused discussions) may have strengthened their implementation by empowering teams not only to identify and respond to challenges but also to recognize and leverage their strengths. Most teams who engaged in our process have adopted CW priorities into their practice and have made it part of their routine care to support care delivery – this is the very definition of sustainability^12,19,20. Third, bringing teams together in focus groups (at 2 time points) and completing the sustainability survey (at 3 time points) became a reinforcing activity for teams to identify and respond to their implementation challenges. In fact, systematically bringing teams together and facilitating discussions may have become an intervention in itself. This highlights a need for a formalized process to engage implementation teams, so they have a forum to discuss and work through their challenges. The question remains, how much involvement do teams need to optimize implementation? It’s important to understand implementation teams and their context and to tailor strategies according to their variable needs. Fourth, using experienced facilitators during our qualitative sessions helped participants to understand the underlying concepts of sustainability, and the survey results, to stimulate discussion and prompt ideas, and to encourage teams to apply their new knowledge into practice. Evidence shows that facilitation is becoming an important aspect of implementation science, and has potential to support innovations for optimized impact^20,47. Lastly, ensuring sustainability capacity is needed to scale up innovations²¹. One of our outputs was a conceptual sustainability and scalability guide, which after further validation could increase the potential for any future CW campaigns to be successfully scaled up at other hospital and primary care settings in Ontario.

Our study had some limitations. First, not all teams were at the same implementation stage of their CW initiative, which may have reduced the reliability of our trend analysis; mid/late implementers had different experiences from early implementers. However, we used focus group data to explain divergent trends including the decreased implementation scores we observed for two of the 15 teams. Second, we observed variable rates of attrition and participation both within and across teams, which may have diminished the representativeness of our teams. However, frequent staff changeover (e.g., revolving residents and students, maternity leaves) are a normal part of clinic team operations, so our participation rates are a reflection of that reality. Not surprisingly, hospital teams had a higher attrition rate than FHTs (39% vs. 15%) in part because the nine hospital teams had variable implementation schedules whereas all six FHTs had fully implemented their CW priority (PPI de-prescribing) at the start of our study. Additionally, the governance and operational structure of FHTs were more streamlined; all six FHT teams were governed by one clinic lead and focused on a single CW priority area (i.e., PPI de-prescribing) compared with the nine hospital teams that implemented four different CW priorities (PPI, BUN/Urea, Pre-Op, and IP Echo) with each hospital having their own clinic/administrative leads. Lastly, it is not possible for us to be certain that any improvement observed was due to our study (which could be considered an intervention in itself), or due to any existing supports or other quality improvement efforts in place at the time of our study. However, we made every effort to document each team’s implementation processes and workflows to identify these. In fact, we encouraged teams to implement changes that would allow embedding into their existing processes and other QI initiatives.

Evaluating the sustainability potential of an innovation such as Choosing Wisely is critical to ensuring that they have the best potential for impact. Our work highlights that implementation teams can be empowered to influence their implementation efforts, and to realize positive outcomes for their healthcare services and patients. CW teams gained increased knowledge about the importance of sustainability and were given an opportunity to co-create an action plan to address their challenges. Providing opportunities for a solutions-focused discussion as part of implementation empowers teams to identify and respond to challenges as well as to recognize and leverage their strengths – these can lead to sustained improvements in health care services and patient outcomes over time.

CW: choosing wisely; FHT:family health team; PPI:proton pump inhibitor; CTO:clinical trials Ontario; NYGH:North York General Hospital; MSH:Markham Stouffville Hospital; SJHC:St. Joseph’s Health Centre; SLRHC:Southlake Regional Health Centre; MGH:Michael Garron Hospital; NHS:National Health Service; KT:knowledge translation; GI:gastrointestinal.

Ethics approval and consent to participate

We obtained ethics approval through Clinical Trials Ontario (CTO) for NYGH, Markham Stouffville Hospital (MSH), St. Joseph’s Health Centre (SJHC) and Southlake regional Health Centre (SLRHC). REB approval for Michael Garron Hospital (MGH) was attained through their REB office.

Consent for publication

Written informed consent was obtained by all survey and focus group participants.

Availability of data and material

Data for the survey and focus group studies are in the Appendices. Other data sets from this study are available upon reasonable request from the corresponding author.

Competing interests

The authors have no competing interests to declare.

Funding

This research was supported by a grant from Adopting Research To Improve Care (ARTIC), Health Quality Ontario, Ontario Ministry of Health and Long-Term Care. They had no role in the study’s design, conduct and reporting of this research study.

Authors' contributions

MK and DM conceived the idea; all authors participated in the design and execution of the study. MK, JM, KM, KH, LH, developed the evidence tables and supporting graphs representing the survey and focus group data. MK drafted the manuscript, and all authors read and approved the final manuscript.

Acknowledgements

We acknowledge and thank all individuals across the five Ontario community hospitals and primary care sites who contributed their time to participate in the surveys and focus groups. MK is funded by a Canadian Institutes of Health Research (CIHR) New Investigator Award.

Schwartz AL, Landon BE, Elshaug AG, Chernew ME, McWilliams JM. Measuring Low-Value Care in Medicare. JAMA Intern Med. 2014;174(7):1067–76.
Brownlee S, Chalkidou K, Doust J, et al. Evidence for overuse of medical services around the world. The Lancet. 2017;390:156–68.
Canadian Institute for Health Information. CIHI. Unnecessary care in Canada: technical report. Ottawa, ON: CIHI, 2017.
Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PM, Korevaar DA, et al. Increasing value and reducing waste in biomedical research: who's listening? Lancet (London, England). 2015.
Institute of Medicine (US) Roundtable on Evidence-Based Medicine; Yong PL, Saunders RS, Olsen LA, editors. The healthcare imperative: Lowering costs and improving outcomes: workshop series summary. National Academies Press (US); 2010. Available at: http://www.ncbi.nlm.nih.gov/books/NBK53920/.
Wiltesy Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Impl Sci. 2012;7:17.
Saini V, Brownlee S, Elshaug AG, et al. Addressing overuse and underuse around the world. Lancet. 2017;390:105–7.
Choosing Wisely Canada. http://www.choosingwiselycanada.org/.
Bhatia RS, Levinson W, Shortt S, Pendrith C, Fric-Shamji E, Kallewaard M, Peul W, Veillard J, Elshaug A, Forde I, Kerr EA. Measuring the effect of Choosing Wisely: an integrated framework to assess campaign impact on low-value care. BMJ Qual Saf. 2015;24:523–31.
Levinson W, Kallewaard M, Bhatia RS, Wolfson D, Schortt S, Kerr EA, On behalf of the Choosing Wisely International Working Group. ‘Choosing Wisely’: a growing international campaign. BMJ Qual Saf. 2015;24:167–74.
Grimshaw JM, Patey AM, Kirkham KR, et al. De-implementing wisely: developing the evidence base to reduce low-value care. BMJ Qual Saf. 2020;0:1–9.
Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059–67.
Proctor E, Luke D, Calhoun A, McMillan C, Brownson R, McRary S, Padek M. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Impl Sci. 2015;10:88.
Kerr EA, Kullgren JT, Saini SD. Choosing Wisely: How To Fulfill The Promise In The Next 5 Years. Health Aff. 2017;36(11):2012–18.
Rosenberg A, Agiro A, Gottlieb M, et al. Early trends among seven recommendations from the choosing wisely campaign. JAMA Intern Med. 2015;175:1913–20.
Mafi JN, Parchman M. Low-value care: an intractable global problem with no quick fix. BMJ Publishing Group Ltd, 2018.
Prasad V, Ioannidis JP. Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implement Sci. 2014;9:1–5.
Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2005. p. 429.
Pluye P, Potvin L, Denis J-L. Making public health programs last: conceptualizing sustainability. Eval Program Plann. 2004;27(2):121–33.
Doyle C, Howe C, Woodcock T, Myron R, Pheko K, McNicholas C, et al. Making change last: Applying the NHS Institute for Innovation and improvement sustainability model to healthcare improvement. Implement Sci. 2013;8:127.
Kastner M, Sayal R, Oliver D, Straus SE, Dolovich L. Sustainability and scalability of a volunteer based primary care intervention (Health TAPESTRY): a mixed-methods analysis. BMC Health Services Research. 2017;17:514.
WHO. Nine steps for developing a scaling up strategy 2010. Available at: http://www.who.int/immunization/hpv/deliver/nine_steps_for_developing_a_scalingup_strategy_who_2010.pdf. Accessed in July 2017.
Eaton J, McCay L, Semrau M, Chatterjee S, Baingana F, Araya R, et al. Scale up of services for mental health in low-income countries and middle income countries. Lancet. 2011;378:1592–603.
Tricco AC, Ashoor HM, Cardoso R, MacDonald H, Cogo E, Kastner M, Perrier L, McKibbon A, Grimshaw JM, Straus SE. Sustainability of knowledge translation interventions in healthcare decision-making: a scoping review. Implementation Science. 2016;11:55.
Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;4:CD000259.
Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, Bunger AC. Public health program capacity for sustainability: a new network. Implementation Science. 2013;8:15.
Savaya R, Spiro S, Elran-Barak R. Sustainability of social programs: a comparative case study analysis. Am J Eval. 2008;29:478–93.
Chambers LL. Factors for sustainability of evidence-based practice innovations: Part I. Research and theory for nursing practice. 2015;29(2):89–93.
Chambes DA, Glasgow RE, Strange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implementation Science. 2013;8:117.
Arnold SR, Straus SE. Interventions to improve antibiotic prescribing practices in ambulatory care. Cochrane Database Syst Rev. 2005;2:CD003539.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76.
Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383(9912):176–85.
Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.
Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. American Journal of Evaluation. 2005;26(3):320–47.
Ham C, Kipping R, McLeod H. Redesigning work processes in health care: lessons from the National Health Service. Milbank Q. 2003;81(3):415–39.
Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 2nd ed. Thousand Oaks: Sage Publications; 2011.
Pinnock H, Barwick M, Carpenter C, Eldridge S, Grandes G, Griffiths CJ, Rycroft-Malone J, Meissner P, Murray E, Patel A, Sheikh A. Taylor SJC for the StaRI Group. Standards for Reporting Implementation Studies (StaRI) statement. BMJ. 2017;356:i6795.
Maher L, Gustafson D, Evans A. NHS Sustainability Model. NHS Institute for Innovation and Improvement; 2010. Available at: .
Design effects in the transition to web-based
Dillman DA, Smyth JD. Design effects in the transition to web-based.
surveys. Am J Prev Med. 2007;32(5 Suppl):S90–6.
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2006;3(2):77–101.
Elo S, Kyngas H. The qualitative content analysis process. JAN Research Methodology. 2007;62(1):107–15.
Wendler MC. Triangulation using a meta-matrix. J Adv Nurs. 2001;35(4):521–25.
Miles MB, Huberman AM, Saldana J. Qualitative data analysis: A methods sourcebook. Incorporated: SAGE Publications; 2013.
Rosenberg A, Agiro A, Gottlieb M, et al. Early trnds among seven recommendations from the Choosing Wisely campaign. JAMA Int Med. 2015;175(12):1913–20.
Hong AS, Ross-Degnan D, Zhang F, Wharam JF. Small decline in low-value back imaging associated with ‘Choosing Wisely’ campaign, 2012–2014. Health Aff. 2017;36(4):671–9.
Colla CH, Kinsella EA, Morden NE, Meyers DJ, Rosenthal MB, Sequist TD. Physician perceptions of Choosing Wisely and drivers of overuse. Am J Manag Care. 2016;22(5):337–43.
Cranley LA, Cummings GG, Profetto-McGrath J, Toth F, Estabrooks CA. Facilitation roles and characteristics associated with research use by healthcare professionals: a scoping review. 2017;7:e014384.

Download PDF

Version 1

posted

You are reading this latest preprint version

An Idea Worth Sustaining: Evaluation of the sustainability potential of Choosing Wisely across Ontario Community Hospitals and Family Health Teams

Status:

Version 1

Abstract

Figures

Contributions To The Literature

Background

Methods

Design overview

Recruitment of CW teams

PHASE 1: Baseline Assessment of sustainability (T1)

PHASE 2: Three-month follow-up (T2)

PHASE 3: Final follow-up (T3)

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1