Building Capacity for Implementation – The KT Challenge

doi:10.21203/rs.3.rs-112876/v1

Download PDF

Research

Building Capacity for Implementation – The KT Challenge

https://doi.org/10.21203/rs.3.rs-112876/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

The KT Challenge program supports health care professionals to effectively implement evidence-based practices. Unlike other knowledge translation (KT) programs, this program is grounded in capacity building, focuses on health care professionals (HCPs), and uses a multi-component intervention. This study presents the evaluation of the KT Challenge program to assess the impact on uptake, KT capacity and practice change.

Methods

The evaluation used a mixed-methods, time-series design involving surveys and review of documents such as teams’ final reports. Online surveys collecting both quantitative and qualitative data were deployed at four time points (after both workshops, six months into implementation, and at the end of the two-year funded projects) to measure KT capacity ( knowledge, skills and confidence), and impact on practice change. Qualitative data was analyzed using a general inductive approach and quantitative data was analysed using non-parametric statistics.

Results

Participants reported statistically significant increases in knowledge and confidence across both workshops, at the 6-month mark of their projects, and at the end of their projects. In addition, at the 6-month check-in, practitioners reported statistically significant improvements in their ability to implement practice changes. In the first cohort of the program, of the teams who were able to complete their projects, half were able to show demonstrable practice changes.

Conclusions

The KT Challenge was successful in improving the capacity of HCPs to implement evidence-based practice changes and has begun to show demonstrable improvements in a number of practice areas. The program is relevant to a variety of HCPs working in diverse practice settings and is relatively inexpensive to implement. Like all practice improvement programs in health care settings, a number of challenges emerged stemming from the high turnover of staff and the limited capacity of some practitioners to take on anything beyond direct patient care. Efforts to address these challenges have been added to subsequent cohorts of the KT Challenge program and ongoing evaluation will examine if they are successful. The KT Challenge program has continued to garner great interest among practitioners, even in the midst of dealing with the COVID-19 pandemic, and shows promise for organizations looking for better ways to mobilize knowledge to improve patient care and empower staff. This study contributes to the implementation science literature by providing a description and evaluation of a new model for embedding KT practice skills in health care settings.

Health Economics & Outcomes Research

Capacity building

Evidence-based practice

Knowledge translation

Health care professionals

Implementation science

Disseminates information about a promising and innovative KT capacity-building program/model
Characterizes unique elements of the program including its focus on health care professionals who have responsibility for implementing practice changes, a multi-component support model, low cost, and applicability to a variety of professionals in diverse practice settings.
Provides evidence from early evaluation of improvements in knowledge, confidence, and ability to implement practice change

Knowledge Translation (KT) initiatives and training programs have proliferated in recent years, as the health system moves towards more evidence-based practice, in order to reduce the gap between the creation of evidence and its implementation into practice (1, 2, 3). Alarmingly, up to 70% of all organizational improvement efforts fail (4). Key barriers include lack of knowledge and skills, lack of time, competing priorities of organizations, and lack of leadership endorsement (5, 6, 7). Important enablers include health care professionals’ (HCPs) positive beliefs about the consequences of participation in training programs, combined with expert guidance and organizational support (8). HCPs want to provide the best care possible, but many lack the knowledge and skills to move evidence into practice in effective ways. A 2012 survey of health care providers, administrators and researchers revealed that nearly 80% of respondents wanted to improve their knowledge and skills related to implementation (7). Research has demonstrated that high-quality implementation strategies are associated with successful implementation (1), thus supporting the need to offer KT and implementation capacity building, training and support to clinicians. A recent systematic review stressed the importance of increasing the number and reach of training opportunities to address the lack of dissemination and implementation training options (9).

Providing KT training takes dedicated effort from health care organizations. Much of the research conducted on how to close the evidence-to-practice gap has been led by and/or focused on academic researchers with minimal involvement from those who provide or use health services (10). First-generation interventions used co-creation models. For example, a program in the United Kingdom used a partnership approach to KT and reported that implementation projects co-produced by researchers and end users were more likely to successfully adapt new evidence into practice, and the co-production experience encouraged future collaboration between the parties (10). The challenge presented by these models is that the impetus for practice change usually resides within the researchers and not the clinicians. Researchers and health care leaders have called for training that builds capacity amongst those working at the point of care (11, 12, 13), who often have the responsibility to implement, but do not have adequate skills and support to do so effectively. Involving those who use research knowledge in efforts to implement the knowledge has multiple benefits, including enhancing the effectiveness of implementation efforts (10). Eames and colleagues, for example, found changes in clinician-reported behaviour following participation in their KT capacity-building intervention for occupational therapists, as well as changes in the culture to one in which clinicians engaged in KT as part of their clinical practice (6). A longitudinal evaluation of a program for implementers, called Practicing KT, showed increased use of, knowledge of, and self-efficacy in KT among those who completed the program (5).

Many different programs exist to increase KT capacity, including some that offer training in implementation skills, but there is lack of information on their effectiveness and on the sustainability of the practice changes supported through the programs. Programs that use active forms of learning to promote the acquisition of KT skills have been shown to be effective in transferring knowledge, but there is scant evidence of sustained practice change resulting from these sessions (1, 5, 14, 15). KT scholars have acknowledged the need for longitudinal evaluation of KT capacity-building initiatives to assess sustainability of outcomes from these programs (5, 11).

Armed with these findings, we developed an implementation support program for HCPs, aimed at building capacity to move research evidence into practice. The program, called the “KT Challenge,” was modeled on a similar research capacity-building initiative for clinicians. That program, in place at our organizations for more than 10 years, has been shown to be effective in supporting practice changes, enhancing evidence-based practice and increasing interest in research engagement, including increased interest in graduate school (16, 17). The KT Challenge program was implemented at two large health organizations in British Columbia, Canada, and offered to all HCPs employed at the organizations. We report an evaluation of the KT Challenge program with respect to uptake by HCPs, impact on KT capacity (knowledge, skills, and confidence) of HCPs, and impact on evidence-based practice changes.

Overview of the KT Challenge program

The KT Challenge program is a multi-component implementation support program that involves training, funding, peer review, and mentorship. The program is coordinated by program leads at each organization. The first step in the program involves HCPs identifying an evidence-based practice change they wish to implement and forming an implementation team. Teams submit a brief letter of intent (LOI), which is reviewed by an advisory committee to ensure that the practice change is identified, needed, and has an adequate evidence base. Feedback is provided and most teams are invited to attend two half-day workshops at which they learn how to develop an implementation and evaluation plan to support the uptake of their proposed practice change. As well, the team identifies a mentor, or consults with the program leads to find a mentor. The final implementation proposals undergo peer-review by an advisory committee composed of KT experts, HCPs, and Patient-Family Partners, who decide whether proposals will be funded. The key components of the KT Challenge program are briefly described below.

LOI: In the LOI, teams identify the practice change they want to implement, document the need for this change in their practice context, and summarize evidence of its effectiveness. LOIs also require the signature of a manager to ensure management support and endorsement, and the identification of team members. The LOIs are formatively reviewed and, in keeping with the capacity-building approach of this program, revisions are suggested when required. Some teams are screened out of the program at this point if their proposed LOI does not identify an evidence-based practice change (e.g., intention to identify a best practice and require some type of effectiveness research, rather than implementation support) but most teams are given feedback and invited into the next stage.

Workshops: Teams attend two half-day workshops. The first workshop focuses on developing an implementation plan and the second covers evaluating the effectiveness of the implementation and the uptake of the practice change. Topics include implementation theories and frameworks, stakeholder engagement, identifying barriers and facilitators, implementation strategies, and evaluation planning.

Mentorship: Teams are supported to find a mentor within their clinical area to support implementation of the practice change and assist with navigating facilitators and barriers. The mentor is expected to be available as a resource to the team members throughout the project, from proposal development to project completion. Mentors are also invited to participate in the evaluation of the KT Challenge throughout the project.

Online resources: Teams have access to an online learning site where a curated set of readings and resources is posted, to support the development and evaluation of their implementation plans.

Management support: Teams are required to demonstrate support from their clinical managers to conduct the implementation project and dedicate time to it.

Funding: Successful teams are awarded $5,000 to cover costs related to personnel, materials and supplies, stakeholder engagement, and services (e.g., data analysis, transcription, graphic design, etc.)

Program Leads: The KT Challenge is run by a program lead from each organization. The leads promote the program, coordinate the review processes, monitor team progress, support teams to successfully navigate barriers, and lead the evaluation of the program.

Figure 1 outlines the timeline of activities for each cohort. Funded teams are asked to submit quarterly reports to provide an update on their progress, and to share challenges and successes. These reports provide a touch point for the program leads to further support the teams and assist in resolving challenges. Teams also submit final reports that provide information on the impact of their project on the intended practice changes.

Figure 1 -- Timeline for the KT Challenge Program

Evaluation Plan

The evaluation of the KT Challenge used a time-series, mixed-method design to assess the impact of the program on workshop participants and funded team members. The evaluation focused on the program uptake and KT capacity (knowledge, skills, and confidence) by HCPs, and the impact on practice change. Online surveys were used to collect data at four time points: after each of the two workshops, 6 months into implementation, and at the end of their two-year funding. Only team leaders and mentors of funded teams complete the 6-month and end-of-project surveys. The online surveys collect both quantitative and qualitative data on participants’ knowledge, skills, and confidence. Data on the impact of the practice change is gleaned from the final reports submitted by the team leads.

Data Analysis

The qualitative data was analyzed using a general inductive approach, which involved the identification of themes and sub-themes. The quantitative data was analyzed in Excel (Microsoft Corporation), using the Wilcoxon Signed-Ranks Test to compare pre and post scores on self-ratings of knowledge, skills, confidence, and ability. Statistical significance was set at p < 0.05.

Uptake of KT Challenge by practitioners

Four cohorts of participants have taken part in the program, with a fifth cohort recently launched. The first cohort (2016) has completed their projects. To date, 24 teams have been funded, comprising 185 HCPs. Participants have included a wide range of HCPs involving 23 types of practitioners working within a range of practice settings. Figure 2 shows the representation of health care professions. Almost all projects involved inter-disciplinary teams.

Interest in the program has remained steady, with 13 teams participating in Cohort 3 (6 were funded). While 9 teams initially joined Cohort 4, the number of teams dropped significantly when the COVID-19 pandemic struck, and only 2 teams submitted funding proposals. However, Cohort 5 was launched in October 2020, and 13 teams applied to participate, including teams proposing COVID-related projects.

Figure 2 – KT Challenge Program Participants by Profession

KT Capacity

The average survey response rate from the three cohorts was 76% for post-workshop #1 survey, 59% for post-workshop #2 survey, and 100% for the 6-month check-in survey and the end-of-project survey.

Participants reported statistically significant increases in knowledge and confidence at the 6-month mark of their projects, compared with before the initiative (p < 0.05). In addition, at the 6-month check-in, practitioners reported statistically significant increases in their ability to implement practice changes.

Figure 3: Average ratings of KT capacity at baseline compared to 6-month follow up.

Numerous comments were provided on the surveys and in the final reports attesting to the knowledge, skills, and confidence acquired to effectively support implementation:

“Learning how to do KT in a way that would be measurable, effective and sustainable. Thinking about addressing barriers and not simply providing MORE information and MORE education has stuck with me and has impacted how I approach other initiatives and projects.”

“Changing how I think about implementing or addressing any practice change. I am constantly thinking about what REAL barriers to change might be.”

Impact on Practice Change

Data on the impact of the program on practice changes is available from the first cohort only, as funded projects in subsequent cohorts are still in progress. Eight teams were funded in the first cohort, but two were unable to complete their projects due to management changes and shifting priorities within their departments. Of the remaining six teams, three showed demonstrable practice changes across their respective practice areas, giving the program a 50% success rate:

In a cardiac inpatient ward, depression screening (conducted by nurses) increased from none to 75%, and follow-up discussions with primary care providers were recorded at 36%.
In a tertiary hospital setting, malnutrition screening (conducted primarily by dietitians) increased by 50%.
In a hospital-based physical rehabilitation program, 67% of physical therapists (PTs) and occupational therapists (OTs) reported increased uptake of the Canadian Stroke Best Practice Recommendations.

Three teams completed their projects but were not able to demonstrate measurable practice change. One team conducted a qualitative evaluation that showed intent by participants to take up the new practice, but actual practice change was not assessed. The other two teams measured increases in awareness of the practice change (new guidelines in both cases) but did not assess the degree to which HCPs followed the guidelines.

Cohort 2 (n = 9 teams) and Cohort 3 (n = 6 teams) are currently completing their projects, with promising results, including one team that has significantly increased the percentage of patients with spinal cord injury receiving functional electrical stimulation treatment. Another team has garnered strong interest in their topic – treating problematic methamphetamine use – and has been approached by British Columbia Ministry of Health policymakers as well as staff from the United States Office of National Drug Control Policy, wanting information on their evidence summary and their implementation plan.

Reported Challenges

Despite the positive impacts on practitioners’ KT capacity and demonstrable practice changes, many teams experienced challenges. The most common challenges included:

Team member turnover
Manager turnover
Gaps in communication with mentor
Projects taking more time than anticipated, requiring leads to use unpaid time, and
Lack of support from key stakeholders for the practice change.

These challenges are common to most KT programs (5,6,15). In response to these challenges, the following steps were taken to improve the program for future cohorts:

Program leads provide feedback and suggestions to teams in response to their quarterly reports
Workshops and program documents were revised to specifically highlight the need for the project evaluation to collect data on the uptake of the practice changes
Program leads undertook more promotion of the program within their organizations to strengthen management’s commitment to the funded teams.

The evaluation of the KT Challenge program demonstrates that a multi-faceted implementation support program for HCPs is effective at moving evidence into practice and can be conducted within a modest budget ($5,000 per team). The steady numbers of applicants for the KT Challenge program each year indicate strong and sustained interest in improving patient care and building capacity for HCPs to lead practice improvements. In the most recent cohort (October 2020), in the midst of the global pandemic, 13 teams submitted LOIs to join the program.

Among challenges faced by teams, lack of time, changes in personnel, and shifting work priorities were the most commonly cited barriers. One team leader described that the KT project was “…one of many things on the go for all team members, so the feasibility of keeping the momentum going all the time was just not there.” In the first cohort of eight funded teams, two were unable to complete their projects due to personnel and manager changes. However, even the teams who were unable to complete, or whose projects did not lead to measurable practice change, agreed that participation in the program was beneficial: “Despite the challenges in trying to carry out the project, it was a good opportunity to move forward a KT initiative…”

We had hoped to see all projects measure demonstrable practice change arising from their interventions, but given the many barriers to practice improvements, it is unrealistic to expect 100% success. As Durlak and Dupre have noted, “Expecting perfect or near-perfect implementation is unrealistic. Positive results have often been obtained with levels around 60%; few studies have attained levels greater than 80%. No study has documented 100% implementation for all providers.” (18, p. 331). In comparison, 50% of our completed teams reported evidence-based practice changes. This provides support that the KT Challenge program is effective for implementing practice changes. Furthermore, the 185 HCPs who participated in the program gained knowledge and confidence in KT practice skills, as well as improved ability to implement practice change, and this increase in the number of skilled KT practitioners working within our organizations, is a marker of success.

We adapted the KT Challenge program to include a stronger patient-oriented approach, to enhance relevance of the projects to patients and families, and to improve dissemination of their findings to audiences including patents and families. Beginning with the 2019-2020 cohort, all KT Challenge teams were required to include a Patient-Family Partner on their implementation team, and all funding proposals are now reviewed by a panel that includes Patient-Family Partners.

The finding that the KT Challenge success rate is higher than reported for most other implementation programs leads us to conclude that the KT Challenge is effective. Further, the leaders of the teams who demonstrated practice changes were enthusiastic about their projects’ achievements. Team leaders also commented on their enhanced KT skills, “It has given me knowledge of how to structure practice change, and also how to work with frontline staff to promote this change.”

The evaluation findings show that the KT Challenge has been very effective at increasing capacity in KT skills and providing information on areas where the program can be improved. We have carefully reviewed evaluation comments from each cohort in the KT Challenge and have made adjustments to the program based on feedback from participants, offering progressively improved support with each cohort. We hope this support will lead to a higher percentage of teams demonstrating measurable practice changes.

The KT Challenge program represents a promising strategy that can be applied across a variety of clinical settings to support the effective uptake of evidence-based practice. This evaluation demonstrates that clinicians will respond to opportunities for KT training, and enhanced capacity for KT practice skills is achievable with support, and – most importantly – that successful practice change can result from small-scale, mentored and funded KT projects in clinical practice settings. This evaluation study contributes to the implementation science literature by providing a description of a new and effective model for embedding KT practice skills in health care settings.

HCP Health care professional

KT Knowledge Translation

LOI Letter of Intent

OT occupational therapist

PT physiotherapist

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing Interests

The authors declare that they have no competing interests.

Funding

The KT Challenge program is supported by Providence Health Care (PHC) Research Institute and the PHC Professional Practice Office, and Vancouver Coastal Health Research Institute and the Robert H. N. Ho Enhancing Patient Care Fund.

Authors’ Contributions

AB and MS contributed to the overall KT Challenge program design and evaluation. AB and AC oversaw the program management and operations at their respective health organization sites. MS developed the workshop content, delivered the presentations, developed the evaluation, and consulted with the teams. KC, AH, JK, AL, MM, AS, and MES formed the program advisory committee to review project proposals, and inform program decisions. All authors contributed to writing and revising the manuscript.

Acknowledgements

We would like to acknowledge the 185 HCPs who have taken part in the KT Challenge, the numerous Patient-Family Partners who have advised teams and reviewed proposals, and the leaders in our organizations who provide funding for the KT Challenge program

Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, et al. The case for KT: shortening the journey from evidence to effect. BMJ; 2003; 327:33e5.
Madon T, Hofman KJ, Kupfer L, Glass RI. Public health. Implementation science. Science; 2007; 318:1728e9.
McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the US. N Engl J Med; 2003; 348:2635e45
National Health Service Sustainability Model. Downloaded 24 Sept 2020 from: https://webarchive.nationalarchives.gov.uk/20160805122935/http://www.nhsiq.nhs.uk/media/2757778/nhs_sustainability_model_-_february_2010_1_.pdf ;2010, ISBN: 978-1-907045-36-3
Moore J, Rashid S, Park J, Khan S, and Straus S. Longitudinal Evaluation of a Course to Build Core Competencies in Implementation Practice. Implementation Science. 2018; 13:106.
Eames S, Bennett S, Whitehead M, Fleming J, Low, Shanling O, Mickan S, Caldwell E. A pre‐post evaluation of a knowledge translation capacity‐building intervention. Australian Occupational Therapy Journal; 2018; Vol 65(6).
The Michael Smith Foundation for Health Research. Training and resources to support research use: a BC needs assessment; 2012.(downloaded 31 Aug 2020 from https://www.msfhr.org/sites/default/files/KT_Needs_Assessment_FINAL.pdf
Wenke R, Noble C, Weir KA, et al. What influences allied health clinician participation in research in the public hospital setting: a qualitative theory-informed approach. BMJ Open 2020;10:e036183. doi: 10.1136/bmjopen-2019-036183
Davis R and D’Lima. D. Building capacity in dissemination and implementation science: a systematic review of the academic literature on teaching and training initiatives. Implementation Science; 2020;15:97; https://doi.org/10.1186/s13012-020-01051-6).
Heaton J, Day J, Britten N. Inside the 'black box' of a knowledge translation program in applied health research. Qualitative Health Research; 2015; Vol 25(11).
Park JS, Moore JE, Sayal R, Holmes BJ, Scarrow G, Graham ID, Jeffs L, Timmings C, Rashid S, Johnson AM, Straus SE. Evaluation of the "Foundations in Knowledge Translation" training initiative: preparing end users to practice KT. Implement Sci; 2018; ISSN: 1748-5908, Vol. 13 (1).
Thomson D, Brooks S, Nuspl M, Hartling L. Programme theory development and formative evaluation of a provincial knowledge translation unit. Health Res Policy Syst; 2019; ISSN: 1478-4505, Vol. 17 (1). PMID: 30971263.
Bennett S, Whitehead M, Eames S, Fleming J, Low S, Caldwell E. Building capacity for knowledge translation in occupational therapy: learning through participatory action research. BMC Med Educ; 2016; ISSN: 1472-6920, Vol. 16 (1).
Goodenough B, Fleming R, Young M, Burns K, Jones C, Forbes F. Raising awareness of research evidence among health professionals delivering dementia care: Are knowledge translation workshops useful? Gerontology & Geriatrics Education; 2017; Vol 38(4).
Gaglardi A, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci; 2016; 11:38.
1. DOI 10.1186/s13012-016-0399-1.
Black A, Ali S, Baumbusch J, McNamee K, Mackay M. Practice‐Based Nursing Research: Evaluation of clinical and professional impacts from a research training program. Journal of Clinical Nursing; 2019; 1:7. DOI: 10.1111/jocn.14861.
Black A, Balneaves L, Garossino C, Puyat J, Qian H. Promoting Evidence-Based Practice through a Research Training Program for Point-of-Care Clinicians. Journal of Nursing Administration: 2015; 45(1).
Durlak, J, Dupre E. Implementation Matters: A Review of Research on the Inﬂuence of Implementation on Program Outcomes and the Factors Affecting Implementation. Am J Community Psychol; 2008; 41:327–350 DOI 10.1007/s10464-008-9165-0

Download PDF

Version 1

posted

You are reading this latest preprint version

Building Capacity for Implementation – The KT Challenge

Status:

Version 1

Abstract

Figures

Contributions To The Literature

Background

Methods

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1