In context evaluation, the educational needs of the recipients of the Centre’s CPD programme, impediments that inhibit participation in education, and resources that the Centre can utilise were identified. Our online survey results were analysed for educational needs assessment (Table 2). About half of 178 total respondents (49.1%) reported that they chose to participate in educational programmes based on “usefulness to their work.” Therefore, it was important to provide practical CPD programmes. When asked what makes it difficult to apply learning to the work environment, 38.4% chose “difference in hospital system,” and 36.5% chose “lack of manpower and equipment.” This indicates that educational programmes that do not consider the working environments of learners can become obstacles. In addition, 38.1% and 35.3% picked “no one to cover the work while absent” and “inconvenient location,” respectively, and 78.5% of the respondents reported preferring 1 day to 1 night 2 days as an appropriate programme period.
Table 2
2017 Educational needs assessment: Online survey conducted after the completion of the 2017 programme (Top five, listed in order)
Questions/ rank
|
1
|
2
|
3
|
4
|
5
|
Q. What factors do you consider when choosing a course?
|
Usefulness at work (n=135, 49.09%)
|
Location (n=61, 22.18%)
|
Instructors (n=36, 13.9%)
|
Duration
(n=30, 10.91%)
|
Cost
(n=12, 4.36%)
|
Q. What are the barriers to your participation of the course?
|
No one to cover work (n=98, 38.13%)
|
Inconvenient location (n=36, 35.29%)
|
Lack of information (n=48, 18.68%)
|
Low budget for education assistance (n=16, 6.23%)
|
Indifference of management (n=11, 4.28%)
|
Q. What are the challenges when applying course materials in real work environment?
|
Difference in hospital systems (n=98, 38.43%)
|
Lack of manpower and adequate equipment (n=93, 36.47%)
|
Heavy workload
(n=43, 16.86%)
|
Uncooperative colleagues
(n=12, 4.71%)
|
Indifference of management (n=4, 1.57%)
|
Q. What do you think the most optimal duration of the training programme is?
|
1 day
(n=62, 39.24%)
|
2 days 1 night
(n=62, 39.24%)
|
3 days 2 nights (n=24, 15.19%)
|
more than 5 days
(n=4, 2.53%)
|
4 days 3 nights
(n=2, 1.27%)
|
* The total number of respondents was 204, but 178 responses were analysed excluding missing data. Online surveys with structured questionnaires were conducted and self-administered by respondents. |
Similar results were derived using FGIs, which were conducted to achieve an in-depth interpretation of online survey results (Appendix). Many respondents pointed out impediments to participating in the educational programme such as the absence of staff who can cover them at work, and geographical and temporal accessibility. Poor advertisement of educational programmes and a workplace atmosphere that discourages participation in education were also among the responses. These responses clearly indicate that there are many barriers to participation in educational programmes. Instead, increasing the temporal and geographical accessibility of programmes and making education mandatory were suggested as factors that could lead to higher participation. The ultrasound educational programme was the course that respondents most wished to take, because it is the most applicable in primary medical care. However, the ultrasound programme was cancelled several times in 2017 because of lack of enrolment. This shows that the content and delivery of education can prevent learners from enrolling in a programme despite high awareness of its utility, which is problematic.
In consultation with experts, expanding accessibility and providing education content tailored to the roles played by public health institutions in the community were highlighted. In particular, huge gaps were seen in the actual clinical environment and patient groups between the learners and the university hospital where the educational program was outsourced. Based on these results of the online survey, FGIs, and expert consultations, the ability to quickly identify emergent patients or to treat elderly patients with chronic conditions were especially important in the learners’ clinical environments. The location of the programme and programme length were also critical.
The abdominal and thoracic ultrasonography programme was determined to require the most changes in needs assessment because of the biggest gap between the willingness to attend the programme and actual participation rate. Moreover, the Centre potentially can be held to account for the cancelled programmes if poorly planned during the inspection of government administration. Therefore, we reviewed records from the 2017 educational programme. Initially, the Centre planned to deliver 3 ultrasonography courses for a total of 10 classes. However, only 6 classes were completed, a potential issue of accountability. Financial support by the Ministry of Health and Education and administrative support by the Centre were secured for the programme.
Through input evaluation, we identified the best alternative that satisfied all pre-selected criteria. After consultation with external experts and workshops with stakeholders, responsiveness to priority system needs, potential effectiveness, fit with existing services, affordability, and administrative feasibility were selected as the criteria. The abdominal and thoracic ultrasound programme was maintained in 2018 because it met the priority system needs, enabling quick screening of emergency patients. However, it was also clear that the programme required improvement. We selected educational material and providers through open competitive bidding through a targeted outsourcing strategy for the abdominal and thoracic ultrasound course only, a strategy that was determined to be being superior to the outsourcing strategy used in 2017 when a total of 17 CPD programmes were outsourced en bloc to a single contractor. While the Centre provided administrative support, it selected one academic society in a related field that best fulfilled the criteria.
There was one academic society which fulfilled the criteria of responsiveness to priority system needs and fit with existing services by setting the educational goal as differentiation of emergency diseases and achievement of skills needed for quick referral in primary care. It also met the criteria of potential effectiveness by establishing a practice-centric educational method. By shortening the educational period from 4 nights 5 days in 2017 to 1 night 2 days, we satisfied the criteria of potential effectiveness and fit with existing services. The educational programme described in the initial proposal was affordable and administratively feasible. For these reasons, the academic society was selected as the provider of educational content. Also, considering the demand for better geographical accessibility, we decided to provide education not only in Seoul, but also in other areas.
After adopting these changes, the revised abdominal and thoracic ultrasound programme was offered in 2018. It went as planned, and even had to be expanded due to increased demand. The initial goal was to educate a total of 24 clinicians in 2 sessions of 12 students each. However, as the programme was more accessible than the 2017 programme in terms of time and location, more students signed up than expected. More sessions were scheduled, bringing the programme operation rate to 150% and the education recipient rate to 187%, both being higher than in 2017 (Table 3).
Table 3
Programme plan and outcomes for 2018 programme
|
Plan
|
Outcome
|
Achievement Rate
|
Number of Courses Provided (module)
|
2
|
3
|
150%
|
Number of Participants (n)
|
24
|
45
|
187.5%
|
Duration of Course in Days (D)
|
2 Days
|
2 Days
|
-
|
Duration of Course in Hours (H)
|
16
|
15.8
|
97.8%
|
Two of the authors observed sessions in real time and recorded progress. The learners, instructors, and operators were asked to complete free-answer questionnaires. The data obtained were then analysed during a meeting at the Centre, where concerns were raised that necessary capabilities may not have been properly trained due to the shortened education period. However, the responses from learners during sessions indicated that they were not only satisfied, but also quite confident about their learning. The instructors felt that learners demonstrated the required level of competence. Free-answer questionnaires were collected from all, the instructors, the learners, and the operators. The instructors said that the programme was highly effective, citing intense concentration and enthusiasm among the learners. The learners were satisfied with learning content that could be applied in daily clinical practice and with the diverse case studies they were provided. Many of the learners left comments thanking the instructors and operators. The operators described vibrant interactions between the instructors and the learners.
Goal attainment was examined by looking at post education improvements in the academic achievement (Fig. 1). Academic achievement as measured based on self-efficacy rose from an average of 3.06 out of 10 to an average of 7.9 after the programme. The educational satisfaction levels for 2017 and 2018 programmes (Fig. 2) showed that the 2018 programme scored higher than the 2017 programme in topic satisfaction, teaching method, applicability, and instructor preparedness. In terms of geographic accessibility, the 2018 programme scored lower than the 2017 programme. However, considering that respondents who found the 4 nights and 5 days programme held in Seoul in 2017 too long and too distant did not participate in the first place and only those for whom it was easy to attend participated, the 2017 participants may have been quite satisfied with the geographic accessibility of the programme location. Based on an analysis of per capita education costs, the 2018 programme cost only 84.1% of the total investment in the 2017 program (Fig. 3).
Three to six months after the completion of the program, the recipients were asked if they experienced progress in their performance by applying what they learned in their work. On a scale of 1 to 10, their answers averaged 7.19. One obstetrician-gynaecologist shared a story in a newsletter about how he successfully improved the diagnosis and survival rates of his emergency patients after receiving ultrasound education. All of short-term and long-term effects were examined and reflected in decision-making. An education and training review committee consisting of officials from the Ministry of Health and Welfare, public medical institution directors, medical education experts, and physicians conducted evaluations and made decisions as stakeholders. The review committee pointed out that only a small number of students could receive ultrasound education because of the high cost of the programme. Still, the committee agreed that the content of the programme corresponds to the goals of public medical institutions and the Centre’s core values, and evaluated the modifications carried out in 2018 as effective for meeting educational goals. Accordingly, they decided to maintain the program, and the same programme was provided in 2019 as well.