A Pragmatic Evaluation of a Public Health Knowledge Broker Mentoring Education Program: A Convergent Mixed Methods Study

doi:10.21203/rs.3.rs-819746/v1

Download PDF

Research

A Pragmatic Evaluation of a Public Health Knowledge Broker Mentoring Education Program: A Convergent Mixed Methods Study

https://doi.org/10.21203/rs.3.rs-819746/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Background: Public health professionals are expected to use the best available research and contextual evidence to inform decision making. The National Collaborating Centre for Methods and Tools developed, implemented, and evaluated a Knowledge Broker mentoring program aimed at facilitating organization-wide evidence-informed decision making in ten public health units in Ontario, Canada. The purpose of this study was to pragmatically assess the impact of the program.

Methods: A convergent mixed methods design was used to interpret quantitative results in the context of the qualitative findings. Quantitatively, participants’ knowledge and skills for finding, interpreting, and using evidence were measured before and after program completion via multiple-choice tests. Changes in scores were assessed using paired t-tests. Qualitatively, program participants and management at enrolled public health units were interviewed to explore the effect of program participation. A secondary analysis of these interviews was conducted to determine whether organizations met their evidence use goals set at baseline, and to identify key factors related to implementation of EIDM within the organization.

Results: Post-program scores for knowledge and skills for EIDM were higher compared to pre-program scores (mean difference = 14.0%, 95% CI 8.2%, 19.8%). Organizations met their goals for evidence use to varying degrees. Key themes identified that support an organizational shift to EIDM include definitive plans for participants to share knowledge during and after program completion, embedding evidence into decision making processes, and supportive leadership with organizational investment of time and resources. The location, setting or size of health units was not associated with attainment of EIDM goals; small, rural health units were not at a disadvantage compared to larger, urban health units.

Conclusions: The Knowledge Broker mentoring program effectively increased participants’ knowledge and skill, allowing them to share their learning and support change at their health units. When paired with organizational supports such as supportive leadership and resource investment, this program holds promise as an innovative knowledge translation strategy for organization wide EIDM among public health organizations.

Health Policy

Occupational Medicine

Evidence-informed decision making

knowledge broker

public health

knowledge translation

research evidence

professional development

capacity building

organizational change

mixed methods

This study implemented and evaluated an intensive Knowledge Broker mentoring program to facilitate evidence informed decision making, using an innovative mixed methods design for deeper analysis than is possible with quantitative alone.
Findings highlight key factors for program success are integration of knowledge brokers into activities across the organization, embedding evidence into decision making processes, and supportive leadership with organizational investment of time and resources.
These findings contribute to the broader literature for successful implementation of knowledge broker roles to facilitate evidence-informed decision making, and notably explore successful implementation in smaller, rural organization, previously reported with only limited success.

Evidence-informed decision making (EIDM) in public health integrates the best available evidence from research and context to optimize decision making. Achieving EIDM requires, in part, finding, synthesizing and applying best available evidence from research sources, community data and local contexts, societal and political preferences, and resources (1, 2). EIDM can help maximize population health outcomes given finite public health resources by implementing strategies with known effectiveness (3, 4). The Public Health Agency of Canada outlines core competencies for the workforce including finding, analyzing, and applying evidence from research and community sources (5). In Ontario, Canada, EIDM has been identified as a foundational standard for practice (2).

Despite these expectations, barriers to achieving EIDM persist. Public health practitioners do not universally have the knowledge and skills for finding, appraising, and using evidence in practice; some public health organizations are unable or unwilling to support evidence-informed practice (6–8). Strategies that focus solely on building individuals’ skills for EIDM have been effective in increasing the practitioners’ capacity but have not had organization-wide impact (9–11). To achieve an organizational shift to EIDM, strategies must facilitate organizational change whilst building individual capacity (4, 12, 13).

Reviews of interventions to increase research use in public health have found that strategies should be adapted to the needs and priorities of the organization and individuals (14, 15). Knowledge broker (KB) roles are inherently well-suited to meet this requirement, as the scope of a KB’s role is linked to the organizational context (16). KBs are responsible for knowledge management (obtaining relevant evidence, creating tailored knowledge products, supporting evidence sharing); knowledge linkage and exchange (facilitating collaboration, developing and maintaining networks); and capacity building (helping develop analytic skills, facilitating and evaluating change) (16). In Canada, KBs have long been a component of efforts to support EIDM (17). KBs in Canada are found across diverse contexts, such as facilitating stakeholder relationships for income assistance policies for people who use drugs (18), bridging the research-practice gap for uptake of measurement tools in children’s rehabilitation clinics (19), and partnering with secondary schools to facilitate the uptake of interventions to promote and improve student health (20). While results of studies evaluating KB interventions in public health settings are mixed, KBs show promise in supporting organization wide EIDM (21–25).

This pragmatic study sought to determine the effect of an intensive KB mentoring program delivered by the National Collaborating Centre for Methods and Tools (NCCMT) to facilitate organizational change for EIDM including individual program participant and organizational level use of evidence; perceptions of the value of the KB mentoring program and public health units’ success or challenges in integrating EIDM.

Study Design

A convergent mixed methods design was used to evaluate two sequential cohorts of the KB mentoring Program, with individuals from five public health units participating in each cohort. In the quantitative phase of the study, a single-group pre-test post-test survey was administered to all study participants. This phase sought to measure individual changes in knowledge and skill for EIDM. In the next phase, a descriptive qualitative study (26) investigated why some participating health units were more or less successful in implementing EIDM. This phase explored the organizational impact of the program and included focus groups with senior leadership at each participating health unit as well as individual, in-depth semi-structured interviews with participants and their managers providing opportunity to reflect on the impact of the program on integration of EIDM within the health unit. Findings from the quantitative and qualitative phase were merged and used to evaluate the overall impact of the KB mentoring program and reported following the Good Reporting of a Mixed Methods Study (GRAMMS) checklist (27). Ethics approval was obtained (Hamilton Integrated Research Ethics Board Project #15-016) and participants provided written informed consent.

Sample

Organizations

Any Canadian public health organization involved in front-line service delivery was eligible to participate in the KB mentoring Program. The NCCMT issued an open call through its newsletter and contacted organizations with previously expressed interest in the Program. As a result of these efforts, ten public health organizations confirmed their participation in the program over two sequential cohorts. The first cohort ran from January 2015 to December 2016, and the second cohort ran from January 2017 to June 2018.

Individuals

Organizations identified five-to-six staff members to participate in the program by seeking volunteers or strategic selection. Diverse staff positions were eligible to participate including public health nurses, health promoters, dietitians, public health inspectors, librarians, managers, or others.

Knowledge Broker mentoring program

The program’s objectives were to assess and assist public health organizations to develop organizational capacity for EIDM, as well as to build capacity among selected staff to function as “internal” knowledge brokers in evidence-informed practice. Participants enrolled in the program were given opportunities to develop their knowledge about the EIDM process including the types of evidence used in public health decision making and skills related to searching for, appraising, synthesizing, and applying evidence to practice. The program, co-designed with public health professionals, was also informed by collective decades of experience delivering EIDM education to public health. The NCCMT mentors who delivered the program held relevant graduate degrees and a wealth of experience in public health and EIDM. Mentors were assigned 1-3 organizations in the first cohort, and two organizations in the second cohort. Intervention content was identical for each cohort.

The program began with an organizational assessment with the senior leadership team of each health unit. Assessments identified the leadership team’s EIDM priorities for the program specifically and the organization generally.

Participants took part in a five-day in-person workshop, followed by a three-day workshop at six months, and a two-day workshop at 12 months. Workshops consisted of a small number of didactic lectures, with most time spent in small-group problem-based sessions; content focused on a systematic approach to EIDM (1). Between workshops, participants convened virtually each month to share progress and continue practice-based learning through group critical appraisals of research evidence. Groups at each public health organization met with mentors every 2-4 weeks via teleconference to address emerging questions and strategize organizational change for EIDM. During the final six months of the program, participants completed a rapid review (28). Participants selected synthesis topics with senior management to ensure findings would benefit the public health organization.

Phase 1: Quantitative Study Component

The first phase in this mixed methods evaluation involved a test of knowledge and skills for EIDM administered at baseline and at the end of the 12-month workshop. The validated assessment consisted of ten practice-based multiple-choice questions designed to assess knowledge and skill in the steps of EIDM (29).

Characteristics of each participating health unit and the populations they serve were collected from government websites (30). Participant demographic questionnaires were completed at baseline. Quantitative data were analyzed using IBM SPSS Statistics 26. Descriptive statistics were used for participant demographics. Change in EIDM knowledge and skills scores were analyzed using paired t-tests.

Phase 2: Qualitative Study Component

Following the measurement of pre- and post-program knowledge and skills for EIDM amongst participants, a descriptive qualitative study (Bradshaw et al 2017) (26) was conducted to identify how EIDM was implemented within each health unit. Descriptive qualitative studies explore phenomena or processes from the perspectives of those involved, and findings can often be integrated in a mixed methods analysis (26). The qualitative component of this study allowed for insight into why some participating health units were more or less successful in implementing EIDM.

Organizational priorities for EIDM and program participation were captured using the validated organizational self-assessment tool Is Research Working for You? (31). Priorities set by each participating public health organization during organizational assessments at the program start were used to deductively code interviews (32). There was strong evidence for achieving a goal if it was described by at least three interviewees, including at least one manager; there was some evidence if described by some interviewees but no manager.

Organizational implementation of EIDM was assessed through semi-structured telephone interviews between six- and 18-months following program completion. The third-party interviewer had extensive experience in qualitative evaluations in public health, leadership development and knowledge translation. Interviewees included a purposeful sample of program participants, managers, and senior decision makers.

Interviews probed the program’s organizational impact in terms of overall staff capacity and system and process changes toward realizing the organization’s EIDM goals. Interviews were conducted as part of a general program evaluation and interviewees were not specifically asked to comment on the organization’s progress toward identified goals. Interview questions are included in Appendix 1.

All interviews were recorded and transcribed verbatim with identifying information removed. Data management and coding was facilitated with the use of NVivo 12 Plus. Data were also analyzed using a conventional content analysis approach to describe processes related to EIDM implementation (33). Categories that emerged from this conventional analysis were organized into meaningful clusters (34).

Phase 3: Data Integration

Quantitative and qualitative data were merged in a mixed methods data analysis (35). Change in test scores for knowledge and skills for EIDM were compared to health units’ overall success in EIDM implementation and descriptions of processes described in interviews. Categories identified in the qualitative analysis were linked to each organization’s stated goals, such that achievement of goals across organizations provided insight into factors that impacted EIDM implementation.

Participant characteristics

Public Health Organizations

Ten public health units in the province of Ontario, Canada participated in two cohorts of the program. Participating health units served diverse populations and geographies (Table 1). Two health units invited staff to volunteer, five selected staff to participate, and three used a combination of these approaches.

Individuals

Fifty-five participants took part in the program, ranging from four to eight participants per public health unit. Fifty-three consented to completing pre-tests and post-tests, and 40 completed both pre-tests and post-tests. Five participants left the program due to role changes within their public health unit or leaves of absence and were replaced by new participants. Demographic data were collected from 51 participants (Table 2). Two participants declined to complete the demographic questionnaire, and one was absent when questionnaires were completed. One participant was added to the program following the first workshop based on interest in the program (Figure 1).

EIDM Knowledge and Skill

The mean knowledge and skill score for all completed pre-tests was 61.9% and post-tests was 77.6% (Table 3). For all completed tests, the mean difference in pre-test and post-test scores was +15.7% (p < 0.001). Mean scores were similar for paired tests (63.3% pre-test, 77.3% post-test). The mean difference in paired pre-test and post-test scores was +14.0% (95% CI 8.2%, 19.8%, p<0.001).

Organizational Change for EIDM

Organizational priorities for EIDM were grouped into themes, with specific priorities per organization identified in Table 4. One public health unit has been omitted from this analysis after not providing post-program data. Levels of success in reaching identified EIDM priorities varied (Table 4). In some cases, there was a disconnect between interviewees perceptions at the same public health unit. For example, managers reported that staff were allocated sufficient protected time for EIDM work, while staff noted they needed more time.

st-program data. Levels of success in reaching identified EIDM priorities varied (Table 4). In some cases, there was a disconnect between interviewees perceptions at the same public health unit. For example, managers reported that staff were allocated sufficient protected time for EIDM work, while staff noted they needed more time.

Further inductive analysis of interviews explored factors and themes that may have contributed to public health units’ success and challenges in integrating EIDM.

Increased organizational capacity for EIDM

Each health unit prioritized improving staff skills for EIDM and using research more often. Most prioritized providing staff with time and resources for EIDM. While most reported staff had increased their skills for EIDM, some indicated there was insufficient time and resources to support applying EIDM. Findings reveal strategies for increasing skills, including participants stepping into EIDM support or teaching roles and training additional staff. Reasons why EIDM capacity goals were not achieved included limitations of the program and participants’ transition out of current roles.

Participants consulting or teaching peers

In seven health units, participants entered teaching roles, in some cases, acting as informal consultants within teams or departments. An executive noted that participants, “have, to a certain extent, become champions [of EIDM] within their teams” (Health Unit 7, Senior Executive Z). Some participants were formally assigned with providing orientation and guidance for EIDM. Other participants became informal sources of knowledge and guidance for colleagues, “…[participant] walked our staff through how to apply this approach and so as a result, a brief review was done of the intervention and literature related to it.” (Health Unit 6, Manager V).

At four health units, participants provided formal workshops to the broader organization. One health unit had participants deliver a series of presentations on EIDM, while three others implemented monthly journal clubs where participants led appraisals of journal articles with colleagues.

Overall, management and peers recognized program participants as resources for EIDM. Various strategies allowed participants to share their expertise, reinforcing participants’ learning and developing other staff’s skills.

Limitations of program

Some participants felt the program curriculum did not directly or easily transfer to their work. A participant noted little room to apply EIDM, “our programs are very clearly scripted about what we have to do, and how and why.” (Health Unit 3, Participant O).

Others noted that the program’s focus on research evidence did not equip them for applying other types of evidence,

“When you look at the model about [EIDM] we focused a lot on the research bubble. And so things like the political climate and resources, those other bubbles that are in that model weren’t really touched on but yet I know from my own job that they’re extremely important.” (Health Unit 2, Participant G)

These criticisms of the program curriculum related to its applicability to participants’ work and whether the program adequately prepared participants to apply EIDM in different contexts.

Participant transitions to new roles

Most health units encountered staff turnover during the program. At one health unit, half of the participants had transitioned to new roles by the time the interviews were conducted. It was noted that participants were typically high achievers, which led to their promotion to new roles, “what happened was probably partly because the individuals we selected were high performers and were leaders in and of their own right, but over time they went on to other positions” (Health Unit 1, Manager A). Other participants left roles for personal reasons. Loss of these participants hindered implementation of EIDM, especially as they had become champions for EIDM.

Integration of EIDM into processes

Health units prioritized using evidence more consistently in decision making. Most sought to assess and adapt research evidence more often while fewer prioritized considering quality of evidence, systematically integrating evidence use and holding management accountable for research use.

Thematic analysis uncovered that some health units concretely integrated EIDM into decision-making processes by developing structures, processes or templates to support EIDM, creating or adapting staff positions for EIDM and pursuing additional learning opportunities.

Structures, processes or templates to support EIDM

Six health units described embedding EIDM into reporting templates or processes to prompt providing an evidence base for program planning recommendations. One health unit revised its briefing note template, “We've got a template and a process for doing briefing notes [that includes] what level of literature research are you using” (Health Unit 1, Manager B). Some health units integrated resources supporting EIDM, such as guidelines and tools, into various processes. Others described guides or resource hubs to support all staff at the organization to engage in EIDM. A resource library was shared through a health unit’s internal network to help staff apply EIDM in program planning. These various strategies helped reinforce and remind staff to use EIDM in all processes.

Staff positions dedicated to EIDM

Three health units noted that roles had changed to integrate EIDM work, “because with the training we were given dedicated time, dedicated FTE [full-time equivalent], all of us, and I think the dedicated FTE really helped us”(Health Unit 4, Participant R).

At two health units, additional staff were hired in specialist roles for EIDM support. On the growing number of research analysts at a health unit, “when I first started, I was one of the first [research analysts] and there were only about three at the time, and now I think there's four or five of them” (Health Unit 4, Participant Q). A dedicated specialist role had been created and filled at another health unit, to further educate frontline staff and guide them through rapid review processes.

Staff noted that dedicated time and roles to EIDM not only facilitated implementation, but it reinforced the prioritization of EIDM at the organization.

Lack of plan for participants

Five health units reported specific goals that had not been achieved. Issues included lack of direction for participants,

“We really had to … remind people that we have this training. It wasn’t necessarily, okay, you guys have this training now, we’re going to get you to do this, this and this or this is, kind of, the plan to have it sustainable. It’s, kind of, been us that’s been advocating for the sustainability of it.” (Health Unit 2, Participant H)

Plans for participants at these health units were absent. Newly acquired knowledge and skills were not applied or shared more broadly within the health unit, which led to frustration from participants.

Lack of protected time

Interviewees at six public health units expressed frustration at not dedicating enough time for EIDM practices. Noting that it was difficult to balance staff time to meet requirements while also using evidence,

“I think the challenge is always to sort of protect the time of those people so that they’re able to act as sort of a knowledge broker for the team and that they’re not struggling to sort of meet that need while continuing to do 100 percent of their normal job.” (Health Unit 10, Manager PP)

Interviewees noted that decision makers needed to allocate time dedicated to EIDM work to ensure that EIDM remained a priority and the work sustainable. Similar to lacking a concrete plan for participants to apply their EIDM skills, participants were frustrated at not having the opportunity to apply the knowledge and skills gained from the program.

Changes at the health unit

Some health units faced other factors that limited their focus on EIDM. Three had unexpected large-scale changes, such as in leadership, organizational structure or a merger with another health unit. Changes significantly overshadowed EIDM as a priority, “We've had a lot of change in upper leadership… and just because of all that flux and change we haven't really had a chance to really dig in [to integrating EIDM]” (Health Unit 2, Participant I). Similarly, merging with another health unit delayed progress with EIDM,

“We released our rapid review just as the merger was happening and, you know, we were without phones and fax machines and computers, and the knowledge broker piece was very much lost in that shuffle, and it's just picked up momentum again.” (Health Unit 9, Participant LL).

While the changes described by interviewees were varied, the effect was consistently a lack of focus on EIDM and lower prioritization of EIDM implementation.

Culture

Fewer goals were set for staff attitudes and organizational culture. Many health units prioritized expanding work with external partners. Health units commented on varying extents of culture shift. Culture changes included: leadership support for EIDM, expectations that decisions would use EIDM, acceptance of time for learning and doing EIDM, and peer learning.

Leadership support

Six public heath units’ leadership set expectations for the transition to an EIDM approach. Actions of Medical Officers of Health were specifically cited, “EIDM and the use of the knowledge brokers within the office of the Medical Officer of Health [supported] ongoing involvement and leadership from the MOH office in creating a culture of [EIDM]” (Health Unit 3, Senior Executive K). Noted of another Medical Officer of Health,

“She’s always wanting, you know, to know like where does this come from, you know, and it’s not to kind of, you know, be testing you or whatnot, but it’s really wanting to be grounded in the evidence and that like all the – everything that we do, our practice is to be grounded in evidence.” (Health Unit 7, Manager AA)

Leadership was described as especially impactful for helping generate organization-wide interest in EIDM and an appreciation of its value, beyond participants and their teams.

Expectations for using EIDM

Some health units described a profound shift in expectations for EIDM. Compared to before the program, “I find that staff are more apt to say ‘well, we should look at the research.” (Health Unit 10, Manager OO). At another public health unit, “[EIDM} is definitely the status quo now. It's present everywhere” (Health Unit 4, Participant Q).

At another health unit, not only is there an expectation to use evidence, but to use evidence appropriately, “before, staff would say, ‘We really want to do this, here's the evidence to support it, let's go’, and we're able to say, ‘Actually I think that came in the wrong order, what's our question, let's find the evidence and then let's make a decision’” (Health Unit 4, Participant T).

Overall, these health units described environments where applying EIDM had become an expectation. Staff had become accustomed to preparing evidence and managers consistently required evidence to support decision making.

EIDM Across Public Health Roles

Participants represented a variety of core public health roles, including typically office-based such as policy analysts and community-based positions, such as public health inspectors (Table 2). Participants in diverse roles, described contributing to EIDM implementation. For example, a public health inspector described:

“I think another big way that it’s also impacted me is in my work as a health inspector. I would say … I’ve definitely increased in our peer review literature that I’m looking at and also working with some of the committees that I work on here to disseminate that to the health inspectors.” (Health Unit 7, Participant DD)

Participants who worked in the community were not limited in applying EIDM compared to participants who worked in offices with more consistent computer access. Participating public health inspectors in particular described championing EIDM among their teams.

Acceptance of Time for EIDM

Three public health units allocated time specifically for staff to develop skills for EIDM, and managers were accepting and encouraging of this practice.

“People are just really taking the time [for EIDM] and, supervisors and, managers are providing the time. They're very supportive of giving employees time to learn about this, which is a shift in itself.” (Health Unit 7, Participant BB)

While a lack of protected time to apply EIDM hindered implementation, the provision of time for staff to develop EIDM skills was described as valuable. Acceptance of this time allowed participants and other staff to develop their skills without time pressures.

Lack of staff buy-in

Four public health units found staff on teams that did not have a participant from the KB mentoring program embedded in the team were reluctant to change their processes. For some, this was attributed to the time required for EIDM.

“… there probably still is the tendency for some of the areas to want to find the quick answer … they would be more likely to just search for something that supports [their opinion] as opposed to taking the time to ask it in a more objective way.” (Health Unit 3, Participant M)

Reluctance to adopt EIDM in some departments was due to biases, specifically to participants’ perceived lack of career experience,

“But my team – I had people on my team going ‘I’m not listening to her. She’s only been here for like three years!’ … she does have all this great knowledge, but it just doesn’t work that way.” (Health Unit 10, Manager OO)

In some cases, staff reluctance was directly related to EIDM processes, or to external factors, such as interpersonal relationships.

Lack of EIDM understanding among management

Some participants’ immediate supervisors did not appreciate how EIDM would affect day-to-day work and did not set appropriate expectations. Training of management would be beneficial,

“… engaging a few more at the management level would have resulted in perhaps a few of those glitches that we experienced, like mitigating those a little bit in terms of how we supported more of our frontline staff … and how to best support the team in terms of sharing their learning and building on it more systematically than we were.” (Health Unit 4, Manager P)

There was frustration that workloads and time were not managed effectively. While EIDM work was added, there were no other projects removed from portfolios, leading to participants’ becoming overwhelmed.

Summary of themes for organizational implementation of EIDM

Interviews provided insight into the successes and challenges faced by program participants in implementing EIDM. Thematic analysis uncovered several major factors for the implementation of EIDM on an organizational level, including building capacity for EIDM, integrating EIDM into processes, and organizational culture.

Over two cohorts, ten public health units in Ontario participated in an intensive KB mentoring program. Participants demonstrated increased knowledge and skill for EIDM on written tests and reported changes in their individual practice. Public health units reported varying degrees in meeting their identified EIDM goals. These results show the real-world impact of such a program and factors that influence implementation of EIDM at an organizational level.

Participants achieved higher scores on tests of EIDM knowledge and skills following program completion, consistent with previously reported studies evaluating change in individual capacity for EIDM (10, 11, 24). The small-group, problem-based approach was effective for adult learners by engaging learners in problem solving, building upon learners’ previous experiences, and involving participants in the learning process (36, 37).

Achievement of EIDM implementation varied across organizations. Interviewees at some organizations reported culture shifts to widespread adoption of EIDM, others saw limited implementation. Organizations that demonstrated readiness through strong leadership supportive of resource investment and a culture accepting and enthusiastic for change achieved greater success in realizing EIDM goals. Similar readiness factors have been shown to be important factors for EIDM (4, 12, 22, 24, 38). This study’s findings illustrated that leaders who established clear expectations for EIDM from staff, also helped facilitate resource investments, such as dedicated EIDM support staff and clear plans for program participants to champion EIDM within the organization. These findings are consistent with previous studies in public health settings, where supportive leadership, organizational readiness, and integration of EIDM processes predicted success (22, 39, 40).

Smaller, rural public health units were successful in implementing EIDM, demonstrating that size and location of organizations do not limit EIDM practices. Several of the smallest and most remote participating organizations were successful in achieving their EIDM goals, despite their own expectations of limited success due to relatively fewer available resources. As noted by participants, the impact of program participants was perceived to be greater in smaller organizations than in larger ones. Previous EIDM implementation studies in health-related rural and remote settings identified perceived barriers to EIDM, such as gaps in knowledge, unfavourable attitudes, and lack of resources (41–45). This study’s findings provide an alternative perspective to these barriers and may enable smaller, rural organizations to overcome perceived barriers to EIDM.

Participants’ professional roles did not limit contributions to EIDM practices, consistent with findings reported by others (22, 39). This study found that public health inspectors contributed to advancing EIDM to similar extents as colleagues in other professional roles that are based in offices for their work rather than mobile in the community. Front-line environmental health staff have previously reported tension between mandated practices and emerging evidence, lack of evidence for emerging topics, perception that only medical research evidence should be used and challenges in measuring environmental health-related outcomes (46). In this study, public health inspectors overcame many previously identified barriers for implementing EIDM, suggesting a variety of public health roles can engage in EIDM. Others have noted the important role managers play in supporting public health inspectors to engage in EIDM (46).

One limitation of the KB mentoring Program was the focus on using research evidence, rather than data from other sources such as local context. Further training on the EIDM steps of implementation and evaluation were also identified. This is consistent with reports of EIDM training in public health, noting that staff value evidence from local contexts but lack skills to effectively gather, appraise and synthesize other forms of evidence, and that evaluation is not well-supported (6). To integrate participate feedback, future program iterations should provide a broader curriculum, including finding and assessing non-research evidence, synthesizing multiple types of evidence, adapting evidence to local contexts, and developing and evaluating implementation plans.

The study did not include a control group and as such it is not possible to distinguish the program’s effects from changes that may have occurred without program participation. In future iterations, a time series or stepped wedge design, where organizations are observed prior to intervention, may help further determine the program’s specific effects (47).

KB roles can pose challenges, both for the individuals acting as KBs and the organizations they serve. The KB role is not well-defined, so contributions vary greatly in different contexts (16, 48). Since KBs function between the worlds of evidence and practice, they may not be considered experts or accepted by colleagues in either community (49). To minimize exclusion by public health colleagues this program built capacity of staff already practicing and experienced within public health. In some cases, more junior participants’ attempts to share knowledge were dismissed by senior staff. It is important to carefully consider how staff chosen to participate in such programs will be accepted by peers (22). The use of dedicated KBs may limit organizational capacity for knowledge translation, as evidence bottlenecks can form at the KB level (49). This study found that participants transferred newly acquired knowledge and skills to colleagues. Evidence bottlenecks may be avoided with multiple people within the same organization in the role (48).

While this study offers new insights into a KB mentoring Program to advance EIDM in public health, there are aspects to explore further. Enhanced focus on non-research evidence and implementation should be included in future program iterations. Participation by smaller, rural, or remote organizations should be considered to further explore EIDM implementation in settings typically perceived as lacking capacity. Exploration of EIDM implementation for diverse public health roles will optimize approaches for organization wide EIDM.

This KB mentoring Program was impactful at an individual level, as measured by a test of knowledge and skill and reported practice changes by program participants. Shifting organizational culture to EIDM is complex, but this program shows promise for effective integration of EIDM within organizations. A KB mentoring program can effectively start and accelerate a public health organization’s journey to EIDM. Supportive leadership and invested resources are key factors for integrating EIDM into organizational processes.

EIDM: Evidence-informed decision making

KB: Knowledge broker

NCCMT: National Collaborating Centre for Methods and Tools

Ethics Approval and Consent to Participate

Ethics approval was obtained via the Hamilton Integrated Research Ethics Board (Project #15-016). Participants were informed of the purpose, potential risks and benefits and requirements of the study and provided written informed consent.

Consent for Publication

N/A

Availability of Data and Materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing Interests

The authors declare that they have no competing interests.

Funding

The National Collaborating Centre for Methods and Tools (NCCMT) is hosted by McMaster University and funded by the Public Health Agency of Canada. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada. The funder had no role in the design of the study, collection, analysis, or interpretation of data or in writing the manuscript.

Authors’ Contributions

E.C., D.C. and M.D. designed and implemented the study. M.S. planned and conducted interviews. E.C. and B.D. analysed study results. E.C. wrote the manuscript in consultation with D.C and S.N-S. D.C. and M.D. provided feedback on all drafts and the final version of the manuscript.

Acknowledgements

The authors would like to acknowledge Susan Snelling for her role as a program mentor, Vivian Chen and Nga Dang for their support in data analysis, Jennifer Yost and Kara DeCorby-Watson for their role in developing the KB mentoring program materials, and Susan Jack for her support in preparing this manuscript.

Authors’ Information

Dr. Maureen Dobbins is the Scientific Director of the National Collaborating Centre for Methods and Tools (NCCMT), The NCCMT’s mandate is to support EIDM among public health professionals in Canada. She is also a Professor in the School of Nursing at McMaster University. Dr. Donna Ciliska is a Professor Emeritus in the School of Nursing at McMaster University and holds decades of experience conducting research on EIDM among public health decision makers.

National Collaborating Centre for Methods and Tools. Evidence-Informed Public Health 2021 [Available from: https://www.nccmt.ca/tools/eiph].
Ministry of Health and Long-Term Care. Ontario Public Health Standards: Requirements for Programs, Services, and Accountability. 2018.
Brownson RC, Baker EA, Deshpande AD, Gillespie KN. The Need for Evidence-Based Public Health Evidence-based Public Health. 3rd ed. New York: Oxford University Press; 2018. pp. 1–28.
Brownson RC, Fielding JE, Green LW. Building Capacity for Evidence-Based Public Health: Reconciling the Pulls of Practice and the Push of Research. Annu Rev Public Health. 2018;39:27–53.
Public Health Agency of Canada. Core Competencies for Public Health in Canada. 1 ed2008.
Martin W, Wharf Higgins J, Pauly BB, MacDonald M. "Layers of translation" - evidence literacy in public health practice: a qualitative secondary analysis. BMC Public Health. 2017;17(1):803.
van der Graaf P, Forrest LF, Adams J, Shucksmith J, White M. How do public health professionals view and engage with research? A qualitative interview study and stakeholder workshop engaging public health professionals and researchers. BMC Public Health. 2017;17(1):892.
Peirson L, Ciliska D, Dobbins M, Mowat D. Building capacity for evidence informed decision making in public health: a case study of organizational change. BMC Public Health. 2012;12:137.
Morshed AB, Ballew P, Elliott MB, Haire-Joshu D, Kreuter MW, Brownson RC. Evaluation of an online training for improving self-reported evidence-based decision-making skills in cancer control among public health professionals. Public Health. 2017;152:28–35.
Jones K, Armstrong R, Pettman T, Waters E. Knowledge Translation for researchers: developing training to support public health researchers KTE efforts. J Public Health (Oxf). 2015;37(2):364–6.
Dreisinger M, Leet TL, Baker EA, Gillespie KN, Haas B, Brownson RC. Improving the public health workforce: evaluation of a training course to enhance evidence-based decision making. J Public Health Manag Pract. 2008;14(2):138–43.
Jacob RR, Baker EA, Allen P, Dodson EA, Duggan K, Fields R, et al. Training needs and supports for evidence-based decision making among the public health workforce in the United States. BMC Health Serv Res. 2014;14:564.
Kislov R, Waterman H, Harvey G, Boaden R. Rethinking capacity building for knowledge mobilisation: developing multilevel capabilities in healthcare organisations. Implement Sci. 2014;9:166.
Campbell DM, Moore G. Increasing the use of research in population health policies and programs: a rapid review. Public Health Res Pract. 2018;28(3).
DeCorby-Watson K, Mensah G, Bergeron K, Abdi S, Rempel B, Manson H. Effectiveness of capacity building interventions relevant to public health practice: a systematic review. BMC Public Health. 2018;18(1):684.
Bornbaum CC, Kornas K, Peirson L, Rosella LC. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implement Sci. 2015;10:162.
Canadian Health Services Research Foundation. The Theory and Practice of Knowledge Brokering in Canada’s Health System 2003.
Mendell J, Richardson L. Integrated knowledge translation to strengthen public policy research: a case study from experimental research on income assistance receipt among people who use drugs. BMC Public Health. 2021;21(1):153.
Russell DJ, Rivard LM, Walter SD, Rosenbaum PL, Roxborough L, Cameron D, et al. Using knowledge brokers to facilitate the uptake of pediatric measurement tools into clinical practice: a before-after intervention study. Implement Sci. 2010;5:92.
Brown KM, Elliott SJ, Robertson-Wilson J, Vine MM, Leatherdale ST. Can knowledge exchange support the implementation of a health-promoting schools approach? Perceived outcomes of knowledge exchange in the COMPASS study. BMC Public Health. 2018;18(1):351.
Carter EJ, Hessels A, Cato K, Sun C, Cohen B, Rivera RR, et al. Evaluation of the joint nurse scientist role across academia and practice. Nurs Outlook. 2020;68(3):261–9.
Dobbins M, Traynor RL, Workentine S, Yousefi-Nooraie R, Yost J. Impact of an organization-wide knowledge translation strategy to support evidence-informed public health decision making. BMC Public Health. 2018;18(1):1412.
Langeveld K, Stronks K, Harting J. Use of a knowledge broker to establish healthy public policies in a city district: a developmental evaluation. BMC Public Health. 2016;16:271.
Traynor R, DeCorby K, Dobbins M. Knowledge brokering in public health: a tale of two studies. Public Health. 2014;128(6):533–44.
Yost J, Dobbins M, Traynor R, DeCorby K, Workentine S, Greco L. Tools to support evidence-informed public health decision making. BMC Public Health. 2014;14:728.
Bradshaw C, Atkinson S, Doody O. Employing a Qualitative Description Approach in Health Care Research. Glob Qual Nurs Res. 2017;4:2333393617742282.
O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8.
Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10.
Yost J, Ciliska D, Dobbins M. Evaluating the impact of an intensive education workshop on evidence-informed decision making knowledge, skills, and behaviours: a mixed methods study. BMC Med Educ. 2014;14:13.
Association of Local Public Health Agencies. Public Health Units. [Available from: alphaweb.org/page/PHU2018].
Kothari A, Edwards N, Hamel N, Judd M. Is research working for you? Validating a tool to examine the capacity of health organizations to use research. Implement Sci. 2009;4:46.
Yates J, Leggett T. Qualitative Research: An Introduction. Radiol Technol. 2016;88(2):225–31.
Chapman AL, Hadfield M, Chapman CJ. Qualitative research in healthcare: an introduction to grounded theory using thematic analysis. J R Coll Physicians Edinb. 2015;45(3):201–5.
Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.
Shorten A, Smith J. Mixed methods research: expanding the evidence base. Evid Based Nurs. 2017;20(3):74–5.
Bryan RL, Kreuter MW, Brownson RC. Integrating adult learning principles into training for public health practice. Health Promot Pract. 2009;10(4):557–63.
Lyons P, Bandura RP. Skills needs, integrative pedagogy and case-based instruction. Journal of Workplace Learning. 2020;32(7):473–87.
Ward M, Mowat D. Creating an organizational culture for evidence-informed decision making. Healthc Manage Forum. 2012;25(3):146–50.
Bryant B, Ward M. A strategic approach to workforce development for local public health. Can J Public Health. 2017;108(4):e403-e8.
Cheetham M, Wiseman A, Khazaeli B, Gibson E, Gray P, Van der Graaf P, et al. Embedded research: a promising way to create evidence-informed impact in public health? J Public Health (Oxf). 2018;40(suppl_1):i64–70.
Hisham R, Liew SM, Ng CJ, Mohd Nor K, Osman IF, Ho GJ, et al. Rural Doctors' Views on and Experiences with Evidence-Based Medicine: The FrEEDoM Qualitative Study. PLoS One. 2016;11(3):e0152649.
Mudderman J, Nelson-Brantley HV, Wilson-Sands CL, Brahn P, Graves KL. The Effect of an Evidence-Based Practice Education and Mentoring Program on Increasing Knowledge, Practice, and Attitudes Toward Evidence-Based Practice in a Rural Critical Access Hospital. J Nurs Adm. 2020;50(5):281–6.
Prakash V, Hariohm K, Balaganapathy M. Barriers in implementing evidence-informed health decisions in rural rehabilitation settings: a mixed methods pilot study. J Evid Based Med. 2014;7(3):178–84.
Kitto SC, Peller JC, Villanueva EV, Gruen RL, Smith JA. Rural surgeons' attitudes towards and usage of evidence-based medicine in rural surgical practice. J Eval Clin Pract. 2011;17(4):678–83.
Parsons JE, Merlin TL, Taylor JE, Wilkinson D, Hiller JE. Evidence-based practice in rural and remote clinical practice: where is the evidence? Aust J Rural Health. 2003;11(5):242–8.
Powers A, Pelletier T, Ray R, Reynolds A, Howarth C, Dobbins M. Participation in the National Collaborating Centre for Methods and Tools’ Knowledge Broker Mentoring Program: a public health inspector perspective. Environmental Health Review. 2021;64(1).
Miller CJ, Smith SN, Pugatch M. Experimental and quasi-experimental designs in implementation research. Psychiatry Res. 2020;283:112452.
Lomas J. The in-between world of knowledge brokering. BMJ. 2007;334(7585):129–32.
Kislov R, Wilson P, Boaden R. The 'dark side' of knowledge brokering. J Health Serv Res Policy. 2017;22(2):107–12.

Due to technical limitations, table 1-4 is only available as a download in the Supplemental Files section.

Tables.pdf
Table 1-4
ClarkGRAMMSChecklist20210816.docx
ClarkSupplementaryMaterialsAppendix120210816.docx
Appendix 1

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

A Pragmatic Evaluation of a Public Health Knowledge Broker Mentoring Education Program: A Convergent Mixed Methods Study

Archived Versions:

Version 1

Abstract

Figures

Contributions To The Literature

Introduction

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Supplementary Files

Archived Versions:

Version 1