Based on the Logic model as a roadmap, we designed and implemented the CERTAIN courses according to the following sections (Figure 1).
Inputs
Firstly, we conducted a need assessment for various study sites, including the local resource, support, and faculty, before launch the education program.
(1) Resources and Program Infrastructure
We evaluated the essential education resources of leaner and their institution. Online education resources (literature links, professional society website, videos, etc.) were provided to ensure course participants would receive up to high-quality continuing medical education with minimal expenses.
(2) Technology to Support Program Management
Technology support was essential for project development. We used Zoom (Zoom Video Communication, Inc) as the secure, HIPPA compatible video conferencing solution for our education program. The CERTAIN course also used several advanced tools for program management: Google Drive (file sharing), Airtable (learner and faculty management), Articulate (Remote simulation), Trello (project management), YouTube, Blackboard and Ethos (Learning Management System), Time zone converter. By leverage those digital technologies, the CERTAIN course could be initiated and implemented from anywhere with a high-speed internet connection to learners around the world[11, 12].
(3) Financial Support
Financial resources were necessary to support the program operation. The seed funding for the CERTAIN project was supported by the Mayo Clinic Education grant, following by Chest Foundation, Laerdal Foundation, WHO funding during different stages of development and dissemination. Those funding are primary to support the core program operation and coordination. We are also encouraging partner hospitals to apply for grants in local regions to support local operations.
(4) Faculty
The faculty pool was the foundation on which the CERTAIN educational program could continue developing and implementing globally. The primary faculties came from original CERTAIN investigators and further expanded to multidisciplinary team members (physicians, fellows, residents, nurses, pharmacists, respiratory therapists, etc.). In addition, international CERTAIN investigators from 15 countries joined us as local faculty when we conducted international courses in their country. More than 20 clinical fellows, research scientists, and research fellows were part of the CERTAIN team during their training and rotation over the years. Many of those fellows came from counties of CERTAIN programs were implemented. Their understanding of the local language and culture helped the CERTAIN program's local dissemination. They also helped CERTAIN modification based on the local practices and later shared with other countries' sites. We also offer train-the-trainer faculty development courses to prepare the faculty pool for program expansion.
(5) Culture
With the vision to minimize preventable death, disability and expensive complications in acutely ill patients, during the CERTAIN study, we were able to recruit international centers shared the same mission to create and deliver innovative education program, and disseminate the CERTAIN approach globally, and helping physicians and nurses implement meaningful change in acute care hospitals. This culture alignment has proven to be very powerful and effective to allow the team to overcome many barriers to achieve the goal. It also fosters the CERTAIN learning community worldwide to share experiences during all phases of the learning process. We build a culture of a CERTAIN team that had the feedback from peers, educators, and technology greatly influence learner satisfaction, and it must be harnessed to provide practical learning experiences. We are also building the train-the-trainer course and faculty playbook to reflect the course value of collaboration and teamwork. In order to design and deliver an innovative education program of a checklist-based approach, we also used design thinking and change management principles to develop standard program management to support idea generation to course delivery[13].
Activities (what do I do?)
CERTAIN course series included the four components, considering specific content of the training, development goals of the trainee, preparation of specific implementation, and evaluation of all potential expectations.
(1) Need Assessment
We did a qualitative interview that activity as a needs assessment to help customized our education program to a specific hospital group. By adopting qualitative methodology to be carried out by one-to-one interviews or as issue-directed discussions for getting the needs assessment [14, 15]. Q-sort survey methodology is a systematic method to investigate participants' perspectives who represent different opinions on education needs by having course participants rank and sort a series of statements regarding their interests. [16, 17] Participant responses were analyzed using factor analysis with five basic steps: setting up that definition of the domain of the particular issue, development of the statements, selection of the participants representing different perspectives, Q-sort by participants, and analysis-interpretation.
(2) Curriculum Design
We chose and built the specific curriculum across multiple specialties to match the needs of the local program. Built on the solid preliminary data and state-of-the-art implementation science, a multimodal education program consisting of asynchronous on-demand video curriculum, virtual simulation workshop, and video-enabled in-situ weekly case-based coaching and quality improvement project. The 40-week CERTAIN Programs allow the learning and implementation to occur over time in the local ICU setting.
(3) Professional Development
The CERTAIN education program was created by the Mayo Clinic team based on the clinical experience of multiple medical centers. Furthermore, the local champion is a critical part committed to learning activities that are scholar-driven, manger-enable, and organization-supported. The local champion from those hospitals needs to engage key stakeholders, identify sources of funding, coordinates operation, and monitor the implementation [18]. We communicated with the local champion to lead the implementation of the course through video conference platforms to increase the scholars’ participation, engagement, and social dynamics learning. All learners were encouraged to have an individual development plan working towards a continuous improvement goal. Meanwhile, local champions were expected to own the plan and keep their improvement moving forward.
(4) Scholar Activity
Beyond the main quality improvement study published, we investigated various topics during the entire life cycle of program intake, design, development, testing, training, and evaluation. The study topics covered various areas, including education research, quality improvement, patient/provider survey, and clinical research. Because we were using various techniques, that platform provides reliable data for analysis. We were also able to mentor many research trainees to develop different research projects to pursue scholarship in variable areas. After graduation from our research program, those trainees who move to other areas can further expand the future CERTAIN network.
Outputs (what happens immediately?)
The output items of the educational program had a variable consequence with the continuous deepening and promotion of the CERTAIN project. We measured the output variables of the program for many dimensions, which include patients, clinicians under the Logic framework.
(1) Learner Output
As the output item of scholar professional development, the CERTAIN learner would master the skill to conduct a structural and systematic checklist approach and deliver humane and patient-centered care. We currently have 1119 learners from 36 countries, and 234 hospitals completed a live or online CERTAIN training program with a certificate. We established a core faculty team via different development pathways through diversified course delivery mechanisms (live, international, remote, train-the-trainer, etc.). This mechanism would greatly increase the faculty pool for the continued international expansion in the future.
(2) CERTAIN Main Study
CERTAIN was a real-time electronic decision aid that offers a systematic approach to perform an initial assessment and ongoing evidence-based management of the critically ill[19]. Since 2014, our team had successfully implemented the CERTAIN training program in a network of 36 hospitals from low-middle income countries (www.icertain.org) using web-based remote simulation and coaching. More than 900 physicians and nurses had completed the CERTAIN training program, and >5000 patients had been enrolled in the international clinical trial (ClinicaltTrials.gov NCT01973829). Recently published data had shown that CERTAIN implementation was feasible and was associated with better adherence to basic critical care processes, decreased intensive care unit (ICU) and hospital length of stays (LOS), and improved survival[1, 2].
(3) Auxiliary Study
We had published study results in numerous journals and presented them in various settings (e.g., Society of Critical Care Medicine Annual Meeting, Karolinska Institute-Mayo Annual Meeting, Mayo Clinic Annual Instructional Design Educational Activities meeting). The study data set is also open for investigators for secondary analysis with a different hypothesis.
(4) Education Program:
Our team created a multimodal course consisting of an asynchronous self-paced online curriculum, synchronous simulation workshop, and synchronous video-enabled remote coaching and quality improvement program. The CERTAIN remote coaching is a year-long in situ ICU remote coaching program that offers a longitudinal, weekly, and interactive virtual learning experience with Mayo Clinic critical care experts and program to facilitate and advance critical care quality improvement efforts at the unit or institutional level. The program offers a monthly core critical care curriculum, weekly case-based discussions, journal clubs, along with opportunities to develop, discuss, and collaborate on research and quality improvement projects.
Outcomes (what are my goals?)
We created the core program assessment outcomes to ensure every component can contribute data on key learning outcomes using the standard Kirkpatrick model[20, 21].
(1) Short Term Outcome
a. Online Quiz: We created a series of questions on the learning management system (Blackboard/ Ethos). A quiz is a set of questions that are graded to measure learners' performance and process of study. Quiz results are scored and reviewed by faculty.
b. Engagement: Beyond using Zoom, we used various asynchronous tools to promote learner engagement. In addition to the "icertain.org" website, we shared CERTAIN contents via YouTube, Twitter, message app (Viber, WhatsApp, WeChat). Those platforms have analytics dashboards to guide our engagement efforts.
c. Post Course Survey: Participants received an electronic, anonymous post-survey that collected basic demographic data and feedback on the course content and faculty using a series of questions and a 5-point Likert scale during all CERTAIN programs. Survey questions were developed and refined for clarity and content through iterative pilot testing.
(2) Long Term Outcome:
a. Hospital Safety Culture: Beyond clinical outcome, the CERTAIN remote education program also has the potential to change the perception of the beliefs, attitudes, values, behavioral characteristics of participants and affect staff member attitudes and behaviors [22-24]. Moreover, higher culture of patient safety score has been shown to be associated with better patient outcomes[25]. We are in the process of assessing the safety culture of clinicians in different counties using the validated instrument (SAQ, Safety Attitudes Questionnaire) [22, 26, 27].
b. Patient improvement: Through the CERTAIN research study, compliance of best practices and patient-centered outcomes were measured by local providers. The participants were trained to harness the power of international collaboration to interact and share ideas and solutions with other local champions in the CERTAIN network. The structured, video-assisted tele-education program can effectively improve bedside compliance of evidence-based care and improve patient-centered outcomes in various intensive care units abroad[2]. However, we could not track the changes for those participants who attended only short education programs (live courses). Only those who participated in the longitudinal program (remote coaching) measured the clinical impact before and after the intervention.
Impact (What will be led to by these outcomes?)
(1) Continues Professional Education
Through CERTAIN remote coaching, we can share relevant critical care best practices with other healthcare professionals from all over the world, even during the COVID-19 global pandemic[28]. It can also improve the scholars' professional knowledge and enhance the desire and interest in clinical research. Meanwhile, the program will cultivate advanced educational ability as the output of continuous professional education, which is the foundation for better clinical work and research in the future.
(2) Healthcare Delivery
It will enable clinicians to improve their knowledge and attitude toward high-quality care. Those learning will power them to process improvement during the daily practices based on KSA (Knowledge-Skills-Abilities) model. Through the study and learning during the CERTAIN program, the learner can use the knowledge gained to further advance the field of critical care medicine in their hospital. The CERTAIN learning network is supportive for lifelong learning to transform healthcare delivery.
(3) Patient Care
Through the remote coaching of CERTAIN's structured and systematic checklist mode, the critical care best practices can be implemented more reliable in those hospitals. In addition, through the cultivation and improvement of individuals, implementing CERTAIN coaching in a critical care setting with limited resources resulted in lasting changes in structure and processes. It was associated with improvement in outcomes of critically ill patients and marked cost-savings[2].