Application of Objective Structured Clinical Examinations (OSCE) for The Evaluation of Kampo Medicine Training

doi:10.21203/rs.3.rs-855872/v1

Download PDF

Research Article

Application of Objective Structured Clinical Examinations (OSCE) for The Evaluation of Kampo Medicine Training

https://doi.org/10.21203/rs.3.rs-855872/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 25 Mar, 2022

Read the published version in BMC Medical Education →

You are reading this latest preprint version

Background

The purpose of study was to develop an objective, valid, and reliable assessment method for Kampo medicine using an OSCE for the evaluation of clinical competence in Kampo medicine

Methods

We developed a blueprint followed by a list of 47 assessment items and three task scenarios related to clinical competence in Kampo medicine. An eight-member test committee checked the relevance and pass/fail criteria of the assessment items on a Likert scale. We calculated a content validity index and content validity ratio, and used the Angoff method to set the passing threshold. We trained a total of nine simulated patients with three assigned to each scenario. We conducted an OSCE for 11 candidates with varying medical abilities, and conducted three stations per person, and one evaluator in one room by direct observation evaluated. We used video recordings to test the inter-rater reliability of the three raters. We used the test results to verify the reliability of the evaluation scale.

Results

The inter-rater reliability (Cronbach’s α) was 0.92–1.00, intra-examinee reliability was 0.59–0.95, and the reliability of each task was 0.86, 0.89, and 0.86 for Scenarios 1, 2, and 3, respectively.

Conclusions

We developed a content-valid new OSCE assessment method for Kampo medicine and obtained high inter-rater and test reliabilities. Our findings suggest that this is one of the most reliable and valid evaluation methods for assessing clinical competence in Kampo medicine.

Educational Philosophy and Theory

Internal Medicine

Kampo medicine

OSCE

Kampo education

In recent years, medical education has become increasingly internationalized.[1] For education in Kampo medicine, a type of traditional Japanese medicine, the World Federation for Medical Education has listed “interface with complementary medicine” as an internationally accredited item [2].

In Japan, the Ministry of Education, Culture, Sports, Science and Technology has included Kampo medicine in its Model Core Curriculum for Medical Education [3], and Kampo has been incorporated into the medical education curriculum of 80 universities nationwide. In addition, the Kampo curriculum has been incorporated into the Model Core Curriculum for Dental Education in Japan [4]. This has led to calls for the introduction of Kampo education into not only medical school, but also dental school curriculums.

Based on the results of a survey conducted to understand the educational status of Kampo medicine in the medical schools of 80 universities in Japan [5], Kampo medicine education is typically assessed by means of examinations, reports, and book reports. Although the need for Kampo medicine education is increasing and various universities have proposed such initiatives, to our knowledge, no studies have been conducted on how to evaluate proficiency in the clinical techniques of Kampo medicine, which is usually required.

The diagnostic process of Kampo medicine is unique and differs from that of Western medicine. In the diagnostic process of Kampo medicine, a “Sho” is determined based on a comprehensive assessment that involves characteristic questioning and tongue, abdominal, and pulse examinations, followed by the selection of a Kampo formula based on Kampo medicine theory (see Fig. 1). Therefore, the education system of Kampo medicine has been based on an apprenticeship system for a long time, which is passed down from some limited number of skilled users of Kampo medicine to their disciples. However, there is an increasing international demand for the spread of Kampo education in current medical education. It is necessary to standardize Kampo medicine, an ancient Japanese traditional medicine that has been handed down based on the apprenticeship system, within the framework of medical education, and to promote its spread internationally.

In the field of medical education, simulation tests are known to be effective. A simulation test with a simulated patient is a performance-based test in which the simulated patient plays the role of a patient according to a predetermined scenario [6]. Such a test is highly effective as a method of assessing clinical competence and as such, has been used in objective structured clinical examinations (OSCEs) [7].

Our hypothesis was that an objective evaluation of clinical competence could be introduced to Kampo medicine, a traditional complementary medicine. Our research questions were, first, whether it is possible to transition Kampo medicine education from an apprenticeship system to a standardized evaluation system, and second, whether it is possible to develop an OSCE that can evaluate Kampo medicine competency with high reliability and validity. Therefore, the purpose of this study was to develop an objective, valid, and reliable assessment method for Kampo medicine using an OSCE for the evaluation of clinical competence in Kampo medicine.

Development of an objective structured clinical examination (OSCE)

A blueprint was created by positing a wide range of competencies that would presumably be necessary for learners. This study developed an assessment focusing on the competence of clinical skills within the competency of Kampo medicine. The competencies for Kampo education were based on the competencies proposed by the Japan Council for Kampo Medical Education, which consists of Kampo educators from 82 medical schools [8]. The evaluation criteria were set by an OSCE in response to the blueprint (Supplementary Material 1). A literature survey regarding Kampo medicine was conducted, and a 50-item evaluation chart was created after extracting high-frequency Kampo medicine findings. A test committee (n = 8: six doctors and two dentists; mean ± standard deviation experience working in Kampo medicine physician: 15.1 ± 5.7 years) responded to a rating sheet on a Likert-type scale regarding the relevance of each item, from 1 = questionable to 4 = essential.

A content validity index (CVI) was calculated based on the percentage of total items rated by the experts as either 3 or 4. A CVI of 0.8 or higher was considered valid and indicated the representativeness and clarity of the checklist items [9]. Of the 50 evaluation items, those with a CVI lower than 0.8 were excluded. The median content validity ratio (n = 8) was 1 (range 0.25–1.0) and mean CVI was 0.96. Three of the 50 items had a CVI of less than 0.8. In the study, 47 items were included in the evaluation chart.

Then, in accordance with the blueprint, three problem scenarios corresponding to the Kampo-based concepts of “qi”, “blood”, and “water” were established (Supplementary Material 2). The evaluation charts for the three scenarios with the same level of difficulty (Supplementary Material 3) were developed by the investigators (M.A. and K.Y.) and verified for content validity by a Kampo specialist (T.M.). Each of the investigators had over 10 years of experience working in Kampo medicine. A flowchart detailing the development of the Kampo-OSCE is shown in Fig. 2.

Training of simulated patients according to the scenarios

Fifteen applications for simulated patients were obtained as a result of a public advertisement. Findings for the simulated patients based on a Kampo-based examination were taken by two investigators (M.A. and K.Y.). Nine of the 15 simulated patients consistent with the Kampo medicine findings of the scenario were selected. These nine simulated patients were then divided into three groups based on similarities in their Kampo-based questionnaires and medical examination findings. Next, the scenarios were distributed to the simulated patients, and guidance involving two explanation sessions (6 hours in total) was carried out in advance. The two investigators confirmed that all simulated patients in the same groups could perform according to the scenario, and that the performances among the simulated patients were consistent.

Training highly reliable raters

Problem scenarios and evaluation charts were distributed in advance to six raters, who each had over 8 years of experience working in Kampo medicine. All raters were provided with an explanation of the evaluation method and practiced on evaluation methods.

Passing threshold

We used Angoff methodology to calculate the passing threshold. The test committee took part in a two-session Angoff standard-setting procedure. In the first session, the judges individually used their professional judgment to estimate the score that a minimally competent final-year student should achieve on each tested element of the Kampo-OSCE. In the second session, all test committee members worked toward consensus [10].

Participants

The study participants were recruited through voluntary applications for the Kampo learning program. The participants were medical students, dental students, residents, physicians, and dentists from grades 1 to 5 (Supplementary Material 4). The total number of participants was 11. Three of the 11 participants had more than 8 years of experience working in Kampo medicine. This study was approved by the ethical review board of Kagoshima University (IRB No: 180345-640). Written informed consent was obtained from all participants.

Conducting the Kampo-OSCE trial

The Kampo-OSCE was administered to participants attending the Kampo learning program. Each examinee was assigned three test problems with approximately the same level of difficulty (Supplementary Material 5). The simulated patients played the role of a patient according to the scenario, and the examinee conducted an interview and a Kampo-oriented physical examination, including tongue, abdominal, and pulse examinations, on the simulated patient. One rater evaluated the patient according to the evaluation chart by direct observation. This data will be denoted as “Data 1”. Each test consisted of a 20-minute clinical examination of Kampo medicine findings. The examinee wrote the information on the obtained findings on a descriptive sheet. The descriptive information was then used for post-test feedback. To verify whether the evaluators’ evaluations and so forth were being carried out appropriately, video recordings were made with the consent of the participants.

Verification of the Kampo-OSCE evaluation

To evaluate inter-rater reliability, three raters evaluated one examiner's video while viewing the video recording. Six videos were evaluated. This data will be denoted as “Data 2”.

Cronbach’s α was used to determine the inter-rater reliability and intra-examinee reliability and the reliability of each task. Inter-rater reliability was calculated using Cronbach's α with Data 2. The intra-examinee reliability was calculated using the Cronbach's α between the three scenarios using Data 1. Reliability of the evaluation scale for each scenario was calculated based on the Data 1 between the 11 examinees.

Statistical analysis

SPSS (v. 23.0; IBM Corp., Armonk, NY, USA) was used for all statistical analyses. Cronbach's α was used to calculate intra-class correlation coefficient (ICC). The criteria of Cronbach’s α was 0.00–0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial, 0.81–1.00 = almost perfect [11].

Passing threshold

According to the calculation using Angoff methodology, the test committee selected a passing threshold of 62%.

Inter-rater reliability

The results of the inter-rater reliability obtained from the evaluation using video recordings by three raters were as follows (n = 6). The results of Cronbach’s α were 0.95 (95% confidence interval [CI]: 0.91–0.97), 0.94 (95% CI: 0.90–0.96), 0.96 (95% CI: 0.94–0.98), 0.92 (95% CI: 0.86–0.95), 0.99 (95% CI: 0.98–0.99), and 1.00 (95% CI: 1.00–1.00).

Intra-examinee reliability

The intra-examinee reliability between the three scenarios was shown in Table 1. It shows the results of Cronbach’s α, which was calculated for the three test results of each of the 11 examinees. The values ranged from 0.57 to 0.95.

Table 1

Intra-examinee reliability
Examinee	No. 1	No. 2	No. 3	No. 4	No. 5	No. 6	No. 7	No. 8	No. 9	No. 10	No. 11
Cronbach’s α (95% CI)	0.75 (0.60–0.85)	0.71 (0.52–0.83)	0.80 (0.67–0.88)	0.83 (0.73–0.90)	0.82 (0.71–0.90)	0.57 (0.30–0.75)	0.73 (0.56–0.84)	0.70 (0.52–0.83)	0.81 (0.69–0.89)	0.95 (0.92–0.97)	0.87 (0.79–0.92)
Cronbach’s α was calculated for the three test results for each of the 11 examinee

Reliability of the evaluation scale for each scenario

Reliability of the evaluation scale for each scenario was shown in Table 2. High reliability was found for each of the three problem scenarios (Cronbach’s α = 0.86 [95% CI: 0.79–0.91], 0.89 [95% CI: 0.83–0.93], and 0.85 [95% CI: 0.78–0.91] for Scenarios 1, 2, and 3, respectively).

Table 2

Reliability of the evaluation scale for each scenario
Scenario	Scenario 1	Scenario 2	Scenario 3
Cronbach’s α (95% CI)	0.86 (0.79–0.91)	0.89 (0.83–0.93)	0.85 (0.78–0.91)
Cronbach’s α was calculated based on the data from the 11 examinees who took the tests to assess the internal consistency of the three scenario tests.

Examinees’ score ratios

As described above, the passing threshold was set at 62%. Figure 3 shows the score ratio for each examinee (n = 11) in the three tests.

Our research questions were whether it is possible to shift the education of Kampo medicine from an apprenticeship system to a standardized evaluation system, and whether it is possible to develop an OSCE that can evaluate competence in Kampo medicine with high reliability and validity.

OSCEs with simulated patients are reported to be more effective than role-play education among students [12]. In this study, we developed a new OSCE evaluation method involving simulated patients for Kampo medicine education. In addition, since OSCEs have been reported to be an appropriate means of assessing communication skills in medical education regarding the evaluation of medical interviews and attitudes [13], the OSCE in this study included interviews, Kampo-based medical examinations, and patient attitudes as assessment items. An important implication was that we have succeeded in introducing a reliable and valid OSCE into Kampo medicine education. And it was possible to say that the education of Kampo medicine could transition from an apprenticeship system to the framework of medical education.

Number of stations and content validity

In this study, we developed a high reliable evaluation method focusing on Kampo medicine techniques, which is one of the consensus competencies among Kampo educators in Japanese medical schools. We developed a blueprint of the diagnostic process and basic theory of Kampo. Based on the blueprint, we selected a Kampo formula for a selection task and created three task scenarios. As a result, high reliability was obtained for Scenarios 1, 2, and 3, suggesting that the test is highly reliable.

We set three tasks and conducted the Kampo-OSCE in three stations. Regarding whether this number of stations was sufficient, it has been reported that eight stations is a reasonable compromise as a screening test in terms of high sensitivity (88–89%) and specificity (83–86%).[14] To our knowledge, no reports of OSCEs for Kampo examinations have been published, so it is unclear how many stations are needed for evaluating Kampo. However, the findings of the present study suggest that the reliability of the test with regard to the number of stations may be maintained, even if the number of tasks for the evaluation of experienced candidates is small, while three or more tasks are desirable for the evaluation of inexperienced candidates.

In our study, Cronbach’s α between stations 1, 2, and 3 showed a reliability of 0.59–0.95. One candidate had a low α, suggesting that the reliability may not be maintained across the three tasks for candidates with little experience in Kampo medicine. However, the three tasks were highly reliable for candidates with experience in Kampo. This suggests that for inexperienced examinees, differences between stations may be seen when they are affected by nervousness or not skilled at a task. On the other hand, examinees with lengthy experience in Kampo were less affected by the content and circumstances of the task and demonstrated a certain level of Kampo examination ability.

Validity and reliability

The content validity and reliability of the Kampo-OSCE developed in this study were sufficiently high. A previous systematic review regarding the inter-rater reliability for communication skills assessments noted that the agreement between reviewers was 0.45 [15].

In the present study, we established one evaluator for each station and conducted the evaluations under direct observation. Afterwards, three evaluators conducted individual evaluations while watching video recordings, and the inter-rater reliability of the three evaluators was examined. A high degree of reliability was obtained between the three raters, so it could be said that the raters could evaluate the Kampo consultations similarly.

On the other hand, direct visual and video-recorded assessments differ. The inter-rater reliability in this study was assessed using video recordings, and the results showed high reliability. Previous reports suggest that assessments using clinical imaging and video assessments correlate well with OSCEs [16]. Therefore, the inter-rater reliability was judged to be high for the Kampo-OSCE as well, since high reliability was obtained in the video-based evaluation in this study. It will be necessary to consider whether it is better to evaluate the OSCE under direct visual assessment or to evaluate it after the fact by video recording.

Reliability of the test and its practicality as a pass/fail test

In this study, two evaluators with sufficient experience in Kampo medicine conducted a preliminary examination to improve the validity of the simulated patients, and confirmed that the scenario and simulated patients were consistent in their findings. However, the more experienced examinees felt uncomfortable with the differences between the real patients and scenarios. This has been pointed out previously in a study in which 28 internists took part in an OSCE of a cardiac physical examination using three methods: real patients, cardiac audio-video simulations associated with “normal” standardized patients, and a cardiac patient simulator [17]. The correlation coefficients between participants’ physical examination skills and diagnostic accuracy were 0.39 (P < 0.05) for real patients, 0.29 for standardized patients, and 0.30 for the cardiac patient simulator, and were significantly higher for real than for standardized patients and the audio-video system combination [17], which suggests that the diagnostic accuracy is higher when using real patients; this may be a limitation of using simulated patients for physical examination evaluations. Since the test was conducted on simulated patients, the experienced doctors were more likely to notice the differences between the findings of real and simulated patients, which was confusing.

In this study, we developed a new OSCE assessment method for clinical competence in Kampo medicine, and confirmed its high inter-rater and test reliability. Our results suggest that it is one of the most reliable and valid evaluation methods for assessing clinical competence in Kampo medicine, which is expected to become increasingly important in the future as the introduction of Kampo medicine education progresses. Establishing an objective evaluation method will be a step toward developing Japanese traditional medicine into an international education.

confidence interval

CVI

content validity index

ICC

intra-class correlation coefficient

OSCE

objective structured clinical examinations.

Ethics approval and consent to participate

This study was conducted in accordance with the guidelines provided by the Declaration of Helsinki. This study was approved by the ethical review board of Kagoshima University (IRB No: 180345-640). Written informed consent was obtained from all participants.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed in this study are available from the corresponding author by request.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was supported by JSPS KAKENHI Grant No. JP15K19152 and 18K09949.

Authors’ contributions

MA, HA, HS, SK, KM, KY, TO, TT, HN, TS, KK, YT, RI and NU collected the data, performed the analyses, conducted the literature search, and drafted the manuscript. RA, ST and TM took part in the design of the study and made revisions to the manuscript. MA, AA, and TO conceptualized and designed the study, coordinated the study, helped interpret the findings, and critically reviewed the manuscript. All authors read and approved the final manuscript.

Acknowledgments

We would like to thank the staffs who participated in the OSCE.

Karle H: Global standards and accreditation in medical education: a view from the WFME. Acad Med 2006, 81(12 Suppl):S43-48.
WFME Global Standards for Quality Improvement 2015. World Federation for Medical Education Available from: https://wfme.org/download/wfme-global-standards-for-quality-improvement-pgme-2015/?wpdmdl=884&refresh=60574b19150211616333593. Updated January 1, 2015. Accessed February 19, 2021.
Medical Education Model Core Curriculum Coordination Committee: Model Core Curriculum for Medical Education. Ministry of Educationã࿽࿽Cultureã࿽࿽Sportsã࿽࿽Science and Technology Available from: http://www.mext.go.jp/component/a_menu/education/detail/__icsFiles/afieldfile/2018/06/18/1325989_30.pdf (accessed 9 August 2021).
Model Core Curriculum Revision Coordination Committee: Model Core Curriculum for Dental Education. Ministry of Educationã࿽࿽Cultureã࿽࿽Sportsã࿽࿽Science and Technology Available from: https://www.mext.go.jp/component/a_menu/education/detail/__icsFiles/afieldfile/2018/06/18/1325989_31.pdf (accessed 9 August 2021).
Arai M, Katai S, Muramatsu S, Namiki T, Hanawa T, Izumi S: Current status of Kampo medicine curricula in all Japanese medical schools. BMC Complement Altern Med 2012, 12:207.
Urbina J, Monks SM: Validating Assessment Tools in Simulation. In: StatPearls. edn. Treasure Island (FL); 2021.
Patricio MF, Juliao M, Fareleira F, Carneiro AV: Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach 2013, 35(6):503–514.
Education JCfKM: Essential Lecture on Kampo Medicine: Yodosha; 2020.
McGartland RD, Kempf-Leonard K: Content Validity: Encyclopedia of Social Measurement. New York: Elsevier; 2005.
Morrison H, McNally H, Wylie C, McFaul P, Thompson W: The passing score in the objective structured clinical examination. Medical education 1996, 30(5):345–348.
Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 1977, 33(1):159–174.
Taylor S, Haywood M, Shulruf B: Comparison of effect between simulated patient clinical skill training and student role play on objective structured clinical examination performance outcomes for medical students in Australia. J Educ Eval Health Prof 2019, 16:3.
Comert M, Zill JM, Christalle E, Dirmaier J, Harter M, Scholl I: Assessing Communication Skills of Medical Students in Objective Structured Clinical Examinations (OSCE)--A Systematic Review of Rating Scales. PloS one 2016, 11(3):e0152717.
Currie GP, Sivasubramaniam S, Cleland J: Sequential testing in a high stakes OSCE: Determining number of screening tests. Med Teach 2016, 38(7):708–714.
Setyonugroho W, Kennedy KM, Kropmans TJ: Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review. Patient education and counseling 2015.
Sulaiman ND, Hamdy H: Assessment of clinical competencies using clinical images and videos "CIVA". BMC Med Educ 2013, 13:78.
Hatala R, Issenberg SB, Kassen BO, Cole G, Bacchus CM, Scalese RJ: Assessing the relationship between cardiac physical examination technique and accurate bedside diagnosis during an objective structured clinical examination (OSCE). Acad Med 2007, 82(10 Suppl):S26-29.

No competing interests reported.

Download PDF

Journal Publication

published 25 Mar, 2022

Read the published version in BMC Medical Education →

Editorial decision: Major revision
15 Nov, 2021
Reviews received at journal
05 Nov, 2021
Reviewers agreed at journal
21 Oct, 2021
Reviewers invited by journal
21 Oct, 2021
Editor assigned by journal
21 Oct, 2021
Editor invited by journal
22 Sep, 2021
Submission checks completed at journal
22 Sep, 2021
First submitted to journal
29 Aug, 2021

You are reading this latest preprint version

Application of Objective Structured Clinical Examinations (OSCE) for The Evaluation of Kampo Medicine Training

Status:

Journal Publication

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Method

Development of an objective structured clinical examination (OSCE)

Training of simulated patients according to the scenarios

Training highly reliable raters

Passing threshold

Participants

Conducting the Kampo-OSCE trial

Verification of the Kampo-OSCE evaluation

Statistical analysis

Results

Passing threshold

Inter-rater reliability

Intra-examinee reliability

Reliability of the evaluation scale for each scenario

Examinees’ score ratios

Discussion

Number of stations and content validity

Validity and reliability

Reliability of the test and its practicality as a pass/fail test

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1