Identifying and Supporting Students at Risk of Failing the National Medical Licensure Examination in Japan Using a Predictive Pass Rate

doi:10.21203/rs.3.rs-34475/v1

Download PDF

Research article

Identifying and Supporting Students at Risk of Failing the National Medical Licensure Examination in Japan Using a Predictive Pass Rate

https://doi.org/10.21203/rs.3.rs-34475/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 10 Nov, 2020

Read the published version in BMC Medical Education →

You are reading this older preprint version

Read the latest preprint version →

Background

Students who fail to pass the National Medical Licensure Examination (NMLE) pose a huge problem from the educational standpoint of healthcare professionals. In the present study, we developed a formula of predictive pass rate (PPR)” which reliably predicts medical students who will fail the NMLE in Japan, and provides an adequate academic support for them.

Methods

Six consecutive cohorts of 531 medical students between 2012 and 2017, Gifu University Graduate School of Medicine, were investigated. Using 7 variables before the admission to medical school and 10 variables after admission, we developed a prediction formula to obtain the PPR for the NMLE using logistic regression analysis. In a new cohort of 106 medical students in 2018, we applied the formula for PPR to them to confirm the capability of the PPR and predicted students who will have a strong likelihood of failing the NMLE.

Results

Medical students who passed the NMLE had the following characteristics: younger age at admission, graduates of high schools located in the surrounding area, high scores in the graduation examination and in the comprehensive computer-based test provided by the Common Achievement Test Organization in Japan. However, total score of examination in pre-clinical medical sciences and Pre-CC OSCE score in the 4th year were not correlated with the PPR. Ninety-one out of 531 students had a strong likelihood of failing the NMLE between 2012 and 2017 and 33 of these 91 students failed NMLE. Using the PPR, we predicted 12 out of 106 students will have a strong likelihood of failing the NMLE. Actually, five of these 12 students failed NMLE.

Conclusions

The PPR can be used to predict medical students who have a higher probability of failing the NMLE. This prediction would enable focused support and guidance by faculty members. Prospective and longitudinal studies for larger and different cohorts would be necessary.

Educational Philosophy and Theory

Internal Medicine

National Medical Licensure Examination

Logistic regression analysis

Predicting student failure

Supporting high-risk students

In many countries such as the US, Germany, and Japan, medical students need to pass the National Medical Licensure Examination (NMLE) in order to take a physician’s license, and students who fail to pass the NMLE pose a huge problem from the educational standpoint of healthcare professionals.^1–6 Students failing USMLE Step 1 are often delayed from continuing course work, which affects their graduation and increases costs.⁴ Failing Step 1 can also affect a student’s ability to enter a residency program and in some instances restrict them from applying for residency in specific states.^4,7 Moreover, even if they enter a residency program, their performance in several specialty board examinations was poorer than those who passed the USMLE or the National Board of Medical Examiners (NBME) without failing ^8–12. There is also a concern about repeaters who cannot pass the NMLE and repeatedly take the examination. The pass rate among such repeaters is not sufficient, while the pass rate among those who take the examination the first time is very high: 67% vs 96% in USMLE Step 1 in 2017, and 63.9% vs 93.3% in the NMLE in Japan in 2018.^13,14 Those who fail seem to be much more likely to end up as repeaters.

Most studies in the past decade have primarily focused on the outcome^3,5 or poor performance^2,6 of those who fail USMLE or NMLE. To the best of our knowledge, there are only two studies in the US and Netherlands that have attempted to create models for predicting those who will fail among first-time test takers of Step 1 ⁴, and in the first-year undergraduate medical curriculum¹. However, these studies did not confirm their results using predictors in a new group of students.

Therefore, the goal of the current study was to develop a model that can reliably predict those who would fail the NMLE using a prediction formula for the pass rate of NMLE: predictive pass rate (PPR). The prediction formula was then applied to a new cohort of medical students to identify students who had a high risk and need support.

Current situation of medical education and the number of doctors in Japan

Undergraduate medical education in Japanese medical schools is usually 6 years ^{15, 16}, including 4 years of pre-clinical medical sciences and 2 years of clinical training. Graduates from these medical schools can take the NMLE.

Based on the future prediction of the number of physicians by the Japanese government, the number of medical students per year has been kept around 7,600 in the past two decades (1990th and 2000th ). However, according to Organization for Economic Co-operation and Development (OECD) data, Japan is ranked 28th in terms of the number of practicing doctors among 35 OECD countries ¹⁷, and the number of medical students was also the least among OECD countries¹⁸. Since 2008, the number of medical students has been gradually increased to about 9,400. Therefore, in Japan, those who fail and/or repeat NMLE ultimately affect the Japanese healthcare system due to reasons such as physician shortage and regional maldistribution of doctors.

Participants

To develop a reliable PPR for the NMLE, six consecutive cohorts of 531 students (2012–2017, 6th academic year) of the Gifu University School of Medicine (GUSM) were included. The GUSM is one of 51 public schools that are largely supported by the Japanese government. The cohorts in each academic year comprised 78, 69, 84, 97, 110, and 93 students, respectively. Through a prospective study, a cohort of 106 students in 2018 was investigated to confirm the PPR by predicting which students will have a strong likelihood of failing the NMLE and providing remediation to them (Table 1). Data from the 637 students in 2012–2018 were obtained after the students were anonymized by the academic affairs office of the GUSM. Ethical approval was granted by the GUSM Ethics Committee. Anonymity and confidentiality were guaranteed (date: 11/30/2016, reference number: 28–333).

Table 1

Characteristics of students in the 2012–2017 and 2018 cohorts
Variables		2012–2017 (n = 531)
Variables		Failed (%)	Passed (%)	P-value	Failed (%)	Passed (%)
Gender	Male	34 (8.4)	371 (91.6)	0.0071	4 (5.1)	75 (94.9)
Gender	Female	2 (1.6)	124 (98.4)	0.0071	1 3.7)	26 (96.3)
Age at admission		27.11 ± 7.51	19.67 ± 3.65	< 0.0001	33.00 ± 10.61	19.16 ± 3.15
HS location				< 0.0001
Neighborhood		5 (1.6)	307 (98.4)		1 (1.4)	69 (98.6)
Distant		31 (14.2)	188 (85.8)		4 (11.1)	32 (88.9)
Type of HS				0.3922
Private		19 (7.8)	224 (92.2)		2 (4.4)	43 (95.6)
Public		17 (5.9)	271 (94.1)		3 (4.9)	58 (95.1)
Academic Level of HS		68.91 ± 5.88	69.22 ± 5.67	0.755	66.6 ± 7.77	68.66 ± 5.24
GPA in HS		4.15 ± 0.47	4.46 ± 0.45	< 0.0001	4.53 ± 0.11	4.50 ± 0.45
NCTUA score		83.97 ± 5.16	86.43 ± 3.94	0.00046	86.12 ± 2.11	86.64 ± 3.37
TOEFL score		515.0 ± 28.3	521.7 ± 25.0	0.124	538.8 ± 23.4	520.8 ± 25.1
Academic performance in liberal arts		68.81 ± 19.67	67.86 ± 13.44	0.695	79.40 ± 22.74	88.09 ± 19.44
Total score in basic sciences in the 1st year		73.70 ± 4.14	75.07 ± 4.62	0.085	74.29 ± 5.46	74.26 ± 4.76
Total score in basic biomedical sciences in the 2nd year		66.91 ± 4.45	72.75 ± 6.86	< 0.0001	64.43 ± 1.84	71.59 ± 6.61
Pre-clinical medical sciences in 3rd to 4th year		74.13 ± 4.71	75.99 ± 5.18	0.0370	72.73 ± 3.87	74.70 ± 6.48
CBT-IRT score in the 4th year		47.94 ± 8.98	59.53 ± 10.01	< 0.0001	49.74 ± 5.64	60.55 ± 10.92
Pre-CC OSCE score in the 4th year		4.26 ± 0.42	4.49 ± 0.38	0.00058	3.98 ± 0.37	4.53 ± 0.43
Performance in clinical clerkship in the 5th to 6th year		3.67 ± 0.72	4.01 ± 0.41	< 0.0001	4.43 ± 0.39	4.21 ± 0.52
Graduation examination in the 6th year		−1.33 ± 0.70	0.10 ± 0.95	< 0.0001	−2.24 ± 0.52	0.11 ± 0.89
Holdover during the 1st to 6th year.				0.00018
་		10 (23.8)	32 (76.2)		2 (13.3)	13 (86.7)
-		26 (5.3)	463 (94.7)		3 (3.3)	88 (96.7)
CBT-IRT, Computer-Based Testing with Item Response Theory; HS, high school; NCTUA, National Center Test for University Admissions; Pre-CC OSCE, Pre-Clinical Clerkship Objective Structured Clinical Examination; TOEFL, Test of English as a Foreign Language. Level of HS shows an average of values that quantified the information on the difficulty of entrance examinations in each HS, that is, a higher level means a higher-difficulty entrance examination. The variables above the dotted line show factors before admission, while the ones below the line represent those after admission.

Variables

The dependent variable was “failing to pass the NMLE.” Data were obtained from the department of academic affairs of the GUSM. Explanatory variables included pre-admission variables such as gender, age at admission, location of high school (HS) (neighborhood prefectures including Gifu and Aichi from where about 60% of students enter the GUSM, and distant prefectures including Tokyo, Osaka and other prefectures), type of HS (public/private), academic level of HS (Table 1), HS grade point average (GPA: 5-grade evaluation), and achievement (percentage of correct answers) in the common entrance examination for university (National Center Test for University Admissions, NCTUA). Post-admission variables were Test of English as a Foreign Language (TOEFL) score, academic performance (percentage) in liberal arts, total score (percentage) in basic sciences in the first year, total score (percentage) in basic biomedical sciences in the second year, total score (percentage) in pre-clinical medical sciences in the third and fourth years, score in the nationwide Computer-Based Testing with Item Response Theory (CBT-IRT; which assess pre-clinical education in the fourth year), average score (six-point scale) in the Pre-Clinical Clerkship Objective Structured Clinical Examination (Pre-CC OSCE) in the fourth year, achievement (standardized deviation values) in the graduation examination in the sixth year, performance in clinical clerkship during the fifth and sixth years, and with or without holdover from first to sixth years (Table 1). These data were obtained from the office of academic affairs of the GUSM, and the average value calculated from the data not including the missing value was substituted for the missing value.

Data analysis

First, we used Fisher’s exact tests and independent t-tests to compare demographic data before and after attending university between those who failed and passed the NMLE in the 2012–2017 cohort. Second, for the 2012–2017 cohort, logistic regression predicting the likelihood of passing the NMLE was used to calculate ORs and 95% confidence intervals (95% CIs) after simultaneously controlling for potential confounders. Third, we created a prediction formula to obtain the PPR using logistic regression analysis with all possible models. In order to guarantee the generality, two models using forced entry and stepwise method were created,

To confirm the suitability of this formulas, we used these formulas for the new cohort in 2018 to identify students who had a lower PPR in NMLE (95% or less; strong likelihood to fail the NMLE). SPSS ver. 23.0 Japan for Windows (SPSS Inc., Chicago, IL, USA) was used to perform statistical test. Two-tailed p-values of < 0.05 were considered significant.

Support To A Cohort Of Graduates In 2018

Firstly, nine months before the NMLE, we noticed these risk factors to all students. And then, three months before the NMLE, a group of students with a higher risk to fail were referred to the academic affairs committee. The committee members held an individual face-to-face interview, elaborately reviewed their motivation and preparedness for the NMLE, gave advice on how to study, determined their educational environment (i.e., location, period of time to study, support by family and/or classmates, economical problems), and encouraged and advised them repeatedly.

Characteristics of those who failed and passed the NMLE in the 2012–2017 cohort

Table 1 shows some significant differences in the demographic data and achievements before and after attending university. In terms of pre-admission variables, those who failed the NMLE showed the following characteristics: predominantly male, older at admission, HS in distant prefectures, lower HS GPA, and lower NCTUA score. After attending university, they had significantly lower scores in basic and pre-clinical medical sciences, CBT-IRT, and Pre-CC OSCE and poor performance in clinical clerkship and were predisposed to repeat a year in medical school (Table 1).

Logistic Regression Analysis

Table 2a and Table 2b show the results of logistic regression of variables that predict the likelihood of passing the NMLE. Both results show that medical students who passed the NMLE showed the following characteristics: younger age at admission, HS located in Gifu and Aichi Prefecture, higher scores in CBT-IRT and graduation examination but not the total score in pre-clinical medical sciences, and better performance in clinical clerkship.

Table 2
a: forced entry method
Variables	B	S.E.	Wald chi-square	P-values	Odd ratio (OR)	95% CI
Gender	1.008	0.974	1.071	0.301	2.739	0.406–18.473
Age at admission	-0.133	0.055	5.853	0.0156	0.875	0.786–0.975
HS location	1.603	0.72	4.96	0.0259	4.966	1.212–20.348
Type of HS	-0.77	0.591	1.699	0.192	0.463	0.145–1.474
Academic Level of HS	-0.033	0.051	0.437	0.508	0.967	0.876–1.068
HS GPA	0.266	0.494	0.29	0.59	1.305	0.496–3.434
NCTUA score	0.031	0.063	0.234	0.629	1.031	0.911–1.168
TOEFL score	0.005	0.011	0.184	0.668	1.005	0.984–1.026
Academic performance in liberal arts	-0.021	0.019	1.277	0.258	0.979	0.944–1.016
Basic sciences	0.037	0.084	0.197	0.657	1.038	0.880–1.224
Basic biomedical sciences	0.094	0.091	1.073	0.3	1.099	0.920–1.312
Pre-clinical medical sciences	-0.359	0.111	10.522	0.00118	0.698	0.562–0.867
CBT-IRT	0.161	0.047	11.665	0.00064	1.175	1.071–1.288
Pre-CC OSCE	-0.051	0.7	0.005	0.942	0.95	0.241–3.746
Performance in clinical clerkship	1.424	0.541	6.933	0.00846	4.155	1.439–11.994
Graduation exam	1.728	0.475	13.21	0.00028	5.629	2.217–14.293
Holdover	0.682	0.757	0.81	0.368	1.977	0.448–8.720
AUC	0.970

Table 2
b: stepwise method
Variables	B	S.E.	Wald chi-square	P-values	Odd ratio (OR)	95% CI
Age at admission	-0.155	0.047	10.895	0.00096	0.856	0.781–0.939
HS location	1.576	0.672	5.509	0.0189	4.837	1.297–18.039
Pre-clinical medical sciences	-0.254	0.085	9.012	0.00268	0.775	0.657–0.915
CBT-IRT	0.167	0.04	17.553	0.00003	1.181	1.093–1.277
Performance in clinical clerkship	1.454	0.475	9.537	0.00222	4.279	1.686–10.861
Graduation exam	1.701	0.424	16.089	0.00006	5.478	2.386–12.577
AUC	0.967
AUC, area under the curve; CBT-IRT, Computer-Based Testing with Item Response Theory; HS, high school; NCTUA, National Center Test for University Admissions; Pre-CC OSCE, Pre-Clinical Clerkship Objective Structured Clinical Examination; TOEFL, Test of English as a Foreign Language. Bold letters and digits indicate significance (p < 0.05). The variables above the dotted line show factors before admission, while those below the line were after admission.

Ppr In The Nmle

Given that the PPR in the NMLE is p/100, the logistic regression formula was as follows;

Forced entry method: log{p/(1 − p)} = 7.75849-0.13301 × age at admission + 1.00772 × female + 1.60261 × neighborhood − 0.77012 × type of HS (public) − 0.03346 × level of HS + 0.26591 × HS GPA + 0.03069 × NCTUA score − 0.02104 × score in liberal arts + 0.00456 × TOEFL score + 0.03728 × score in basic sciences in the first year + 0.09396 × score in basic biomedical sciences in the second year − 0.35949 × score in pre-clinical medical sciences from the third to fourth years + 0.16101 × CBT-IRT score − 0.05099 × Pre-CC OSCE score + 1.42424 × performance in clinical clerkship from the fifth to sixth years + 1.72797 × achievement in the graduation examination + 0.68164 × with holdover.

Stepwise method: log{p/(1 − p)} = 11.65386-0.15522 × age at admission + 1.57629 × neighborhood − 0.25435 × score in pre-clinical medical sciences from the third to fourth years + 0.16662 × CBT-IRT score + 1.4537 × performance in clinical clerkship from the fifth to sixth years + 1.70081 × achievement in the graduation examination.

Ninety-one out of 531 students from 2012 to 2017 had a lower PPR for NMLE using the both methods, and actually 34 of 91 students failed the NMLE.

Prediction for the graduates in the 2018 cohort and support

Using the above formulas, we predicted students who will likely fail the NMLE and guided them (Table 3a, 3b). Twelve out of 106 students in 2018 were predicted as having lower PPR (Table 3a), and they were supported by the faculty members. Eleven students predicted as having lower PPR by the stepwise method (Table 3b) were all included in the 12 students predicted in Table 3a. Seven of 12 students passed the NMLE after obtaining support, and five students failed as predicted. Thus the pass rate of NMLE in 2018 was 95.3% (101/106) (national average: 90.1%). As compared to the pass rate of 88.2% in 2017 (82/93 students) (national average: 88.7%), better outcome was obtained. In both models, we could predict all five students who would fail; these were included among the high-risk students.

Table 3
a: forced entry method
Predictive pass rate in the NMLE		Results in the actual NMLE
Range (R)	No. (%)	No. of Students who Passed (%)		No. of Students who Failed (%)
R ≤ 30%	3 (2.8)	0 (0.0)	7(6.6)	3 (2.8)	5 (4.7)
30 < R ≤ 60%	1 (0.9)	1 (0.9)		0 (0.0)
60 < R ≤ 90%	7 (6.6)	5 (4.7)		2 (1.9)
90 < R ≤ 95%	1 (0.9)	1 (0.9)		0 (0.0)
95 < R ≤ 100%	94 (88.6)	94 (88.6)		0 (0.0)
Total	106 (100.0)	101 (95.3)		5 (4.7)

Table 3
b: stepwise method
Predictive pass rate in the NMLE		Results in the actual NMLE
Range (R)	No. (%)	No. of Students who Passed (%)		No. of Students who Failed (%)
R ≤ 30%	2 (1.9)	0 (0.0)	6 (5.7)	2 (1.9)	5 (4.7)
30 < R ≤ 60%	4 (3.8)	2 (1.9)		2 (1.9)
60 < R ≤ 90%	3 (2.8)	2 (1.9)		1 (0.8)
90 < R ≤ 95%	2 (1.9)	2 (1.9)		0 (0.0)
95 < R ≤ 100%	95 (89.6)	95 (89.6)		0 (0.0)
Total	106 (100.0)	101 (95.3)		5 (4.7)
NMLE, National Medical Licensure Examination in Japan. The area of R ≤ 95% means medical students who were judged to need some support prior to graduation and required remediation by the academic affairs committee.

We developed a formula for predicting the pass rate in the NMLE. Using this formula, we evaluated a new cohort of students in 2018 and predicted 12 students who had a higher risk of failing the NMLE. After guidance by faculty members, 7 of the 12 students passed the NMLE and the actual pass rate improved.

Predictors for passing the NMLE

We identified four significant internal predictors for passing the NMLE: 1) total score in pre-clinical medical sciences in the third and fourth years, 2) CBT-IRT score in the fourth year, 3) performance in clinical clerkship in the fifth and sixth years, and 4) score in the graduation examination in the sixth year. We also identified two external predictors: age at admission and HS located in surrounding area.

Among them, CBT is a nationwide examination administered by the Common Achievement Tests Organization¹⁹ for medical students in all Japanese medical schools before clinical clerkship using a computer to estimate the student’s knowledge for the clinical clerkship. CBT corresponds to Step 1 of USMLE, and a number of studies on risk factors and outcome for those who failed Step 1 ^2-5,20,21 and studies investigating Step 1 score as one of the predictors of performance after Step 1.^8-11,22-26 The latter may be correlated to our result for CBT. Most studies including the study by Koenig et al.²²have indicated that a high score in Step 1 is a predictor of success in many fields in the medical profession (i.e., internal medicine, dermatology, ophthalmology, orthopedic surgery, gynecology, and family medicine),^8-11,23-25 with some opposite results ^27,28. Casey et al. ²⁶ noted that the medical college aptitude test (MCAT), Step 1 and Step 2, and subsequent clinical performance parameters correlated with NBME scores across all core clinical clerkships. They also emphasized that Step 1 scores identified students at risk of poor performance in NBME subject examinations, facilitating and supporting implementation of remediation before clinical years.²⁶ Accordingly, it is very reasonable to assume that the score of CBT which is compatible with Step 1 is one of the predictors of passing the NMLE in Japan which is compatible with NBME.

In the present study, additional three other internal predictors for passing the NMLE were also identified: score in pre-clinical medical sciences, performance in clinical clerkship, and graduation examination scores. The NMLE was taken within 3 months after clinical clerkship and graduation examination. The logistic regression analyses in our study showed a negative correlation between the score in pre-clinical medical sciences in the third and fourth academic years and passing the NMLE (Table 2). This result is in conflict with a general thought that the students with higher academic score in preclinical medical science would likely be to pass NMLE. Furthermore, the total scores for pre-clinical medical sciences in the students who failed NMLE in 2012–2017 were actually lower than those who passed NMLE (Table 1). However, when we closely looked into the 36 students who failed, we found that they had older age at admission and better scores in pre-clinical medical sciences but worse performance in the graduation examination. Hence, we hypothesize that older medical students might have insufficient study time because some of them had family or need to work part-time, diminished ability to memorize, or burnout due to longer years of schooling and/or working since they graduated high schools. Further studies are needed to confirm this hypnosis. The four significant internal predictors of passing the NMLE shown in this study can be used to predict those who may fail the NMLE.

Moreover, significant external predictors of passing the NMLE were age at admission and HS located in Gifu and Aichi Prefecture. Using linear regression analysis, Kleshinski et al ²⁰, identified predictors of performance on Step 1 and Step 2 as follows: science GPA, biologic science section of MCAT, college selectivity, race, and age. Furthermore, McDougle et al.³ indicated that the relative risk of first-attempt Step 1 failure for medical school graduates was 3.6 for matriculants aged >22 years (95%CI: 2.0–6.6, p < 0.0001). Consequently, older medical students have a higher risk of failing Step 1, Step 2, and the NMLE. It is unclear why medical students who belonged to a neighborhood HS have better chance of passing the NMLE, and we found no such study on the relationship between performance in medical school and the location of HS or hometown. Students from the neighborhood city/town might be able to receive various kinds of supports from their families physically, economically, and psychologically. Further investigations are required.

Predicting NMLE with data in lower grade

Several studies have predicted the performance of medical students in Step 1 and primarily focused on first-time test takers.^{4,20,24,29,30} Determining the characteristics of a student who will fail Step 1 is challenging⁴ because it is difficult to create models that predict the failure of first-time test takers given the low number of students who fail in most schools.^4,21 Keeping this in mind, Coumarbatch et al. attempted to create models to predict those who will fail among first-time test takers using logistic regression analysis in 256 students from the graduating class of 2008 at Wayne University.⁴ They found that the year-2 standard score and MCAT biological science score were significant predictors of failing and concluded that using internal and external predictors, identifying students at risk of failing Step 1 is possible.⁴ Moreover, they described at-risk groups and current educational intervention strategies. In the current study, the year-2 standard score and MCAT score might correspond to the total score in pre-clinical medical science and the NCTUA (Tables 1 and 2), however, there is a difference of the competencies required and the level of difficulty between MCAT and NCTUA, so it would be reasonable that our results using logistic regression analysis were not consistent with theirs ⁴. More recently, Baars et al. developed a model for the early and reliable prediction of students who fail to pass the first year in the undergraduate medical curriculum within 2 years after starting.¹ However, we cannot directly compare our results and theirs. In the GUSM, the students who failed the NMLE did not have better or worse scores in liberal arts and basic science during their first year in medical school (Tables 1 and 2).

Thus, in the current study, we determined the PPR using several information which can be obtained easily during medical schools, and predicted students who have higher risk to fail NMLE using the PPR for the first time.

The pass rate in NMLE 2018 after support based on the PPR prediction

Before the current study, faculties noticed that some young students with poor performance in the mock examination (ME) may pass the actual NMLE, while the older students with good performance in the ME sometimes failed NMLE, but the reason was unclear. For a new cohort in 2018, we chose students who had lower PPRs in the NMLE (95% or less), indicating a strong likelihood to fail the NMLE, to confirm the validity of the formula (Table 3). The PPR predicted all five students who would fail. Therefore, this result showed that risk analysis from data such as the PPR can enable effective support from multiple points of view, such as the use of MEs. Further prospective studies are needed in other cultural areas, although we need to confirm the validity of the PPR.

First, we cannot directly compare the present and previous studies because of differences in independent variables. Second, our results may be influenced by some differences in the selection of medical students and the medical education system between Japan and other countries. Third, it is not clear may be unclear whether our results can be applied to other Japanese medical schools because there was no report similar to our study and the study period was only 1 year. Therefore, we expect to applicate and verify the knowledge in other Japanese medical schools.

This is the first study that demonstrated six significant predictors for passing the NMLE and the possibility of decreasing the number of students who fail the NMLE prospectively using the PPR. To confirm these results, further studies are needed because there is no similar trial.

AUC

Area Under the Curve

CBT-IRT

Computer-Based Testing with Item Response Theory

GPA

Grade Point Average

GUSM

Gifu University School of Medicine

High School

MCAT

Medical College Aptitude Test

Mock Examination

NBME

National Board of Medical Examiners

NCTUA

National Center Test for University Admissions

NMLE

National Medical Licensure Examination

OECD

Organization for Economic Co-operation and Development

Odds Ratio

PPR

Predictive Pass Rate

Pre-CC OSCE

Pre-Clinical Clerkship Objective Structured Clinical Examination

TOEFL

Test of English as a Foreign Language

USMLE

United States Medical Licensing Examination

Ethics approval and consent to participate

Ethical approval was granted by the GUSM Ethics Committee. Anonymity and confidentiality were guaranteed (date: 11/30/2016, reference number: 28-333).

Consent for publication

(Not applicable)

Availability of data and material

Our data are not on a data repository. The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Only coded data may be shared.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was partly supported by JSPS KAKENHI Grant Number 19K21757.

Authors' contributions

All authors have made contributions to the study and the manuscript. TS made the concept and design of the study. KT managed the data acquisition and statistical analysis. KT and TS performed interpretation of data, as well as the drafting manuscript. YS made the supervision of the study design and modified the manuscript. All authors have read and approved the manuscript.

Acknowledgements

We would like to thank administrators of academic affairs section in Gifu University Graduate School of Medicine for helping authors’ data acquisition.

Baars G, Stijnen T, Splinter T. A model to predict student failure in the first year of the undergraduate medical curriculum. Health Professions Education 2017;3:5-14. doi: 10.1097/ACM.0000000000001938.
Burns ER, Garrett J. Student failures on first-year medical basic science courses and the USMLE step 1: a retrospective study over a 20-year period. Anat Sci Educ. 2015;8:120-125. doi: 10.1002/ase.1462.
McDougle L, Mavis BE, Jeffe DB, et al. Academic and professional career outcomes of medical school graduates who failed USMLE Step 1 on the first attempt. Adv Health Sci Educ Theory Pract. 2013;18:279-289. doi: 10.1007/s10459-012-9371-2.
Coumarbatch J, Robinson L, Thomas R, et al. Strategies for identifying students at risk for USMLE step 1 failure. Fam Med. 2010;42:105-110.
Biskobing DM, Lawson SR, Messmer JM, et al. Study of selected outcomes of medical students who fail USMLE Step 1. Med Educ Online. 2006;11:4589. doi: 10.3402/meo.v11i.4589.
Kies SM, Freund GG. Medical students who decompress during the M-1 year outperform those who fail and repeat it: a study of M-1 students at the University of Illinois College of Medicine at Urbana-Champaign 1988-2000. BMC Med Educ. 2005;5:18. DOI: 10.1186/1472-6920-5-18
Physician Licensing Service. Medical License Requirements by State. https://physicianlicensing.com/resources/state-requirements/ Accessed June 2, 2020.
Fening K, Vander Horst A, Zirwas M. Correlation of USMLE Step 1 scores with performance on dermatology in-training examinations. J Am Acad Dermatol. 2011;64:102-106. doi: 10.1016/j.jaad.2009.12.051.
Swanson DB, Sawhill A, Holtzman KZ, et al. Relationship between performance on part I of the American Board of Orthopaedic Surgery Certifying Examination and Scores on USMLE Steps 1 and 2. Acad Med. 2009;84:S21-24. doi: 10.1097/ACM.0b013e3181b37fd2.
McCaskill QE, Kirk JJ, Barata DM, et al. USMLE step 1 scores as a significant predictor of future board passage in pediatrics. Ambul Pediatr. 2007;7:192-195. DOI: 10.1016/j.ambp.2007.01.002
Myles TD, Henderson RC. Medical licensure examination scores: relationship to obstetrics and gynecology examination scores. Obstet Gynecol. 2002;100:955-958.
Case SM, Swanson DB. Validity of NBME Part I and Part II scores for selection of residents in orthopedic surgery, dermatology, and preventive medicine. Acad Med. 1993;68:S51-56.
2017 Performance Data. http://www.usmle.org/performance-data/default.aspx Accessed June 2, 2020.
Japanese Ministry of Health, Labour and Welfare. Performance Dara for the 112^th National Medical Licensure Examination in Japan (in Japanese). www.mhlw.go.jp/file/05-Shingikai-10803000-Iseikyoku-Ijika/0000197914.pdf Accessed June 2, 2020.
Suzuki Y, Gibbs T, Fujisaki K. Medical education in Japan: a challenge to the healthcare system, Med Teach. 2008;30(9-10):846-850. doi: 10.1080/01421590802298207.
Imafuku R, Saiki T, Suzuki Y. Developing undergraduate research in Japanese Medical Education. Council on Undergraduate Research Quarterly. 2016;37:34-40.
Organization for Economic Co-operation and Development OECD Data: Doctors. https://data.oecd.org/healthres/doctors.htm Accessed June 2, 2020.
Organization for Economic Co-operation and Development OECD Data: Medical graduates. https://data.oecd.org/healthres/medical-graduates.htm Accessed June 2, 2020.
Yamada R. Measuring Quality of Undergraduate Education in Japan: Comparative Perspective in a Knowledge Based Society. Elsevier, 2014, 76-78.
Kleshinski J, Khuder SA, Shapiro JI, et al. Impact of preadmission variables on USMLE step 1 and step 2 performance. Adv Health Sci Educ Theory Pract. 2009;14:69-78. DOI: 10.1007/s10459-007-9087-x.
Zhao X, Oppler S, Dunleavy D, et al. Validity of four approaches of using repeaters MCAT scores in medical school admissions to predict USMLE Step 1 total scores. Acad Med. 2010;85(Suppl):S64-S67. doi: 10.1097/ACM.0b013e3181ed38fc.
Koenig JA, Sireci SG, Wiley A. Evaluating the predictive validity of MCAT scores across diverse applicant groups. Acad Med. 1998;73:1095-1106.
Myles T, Galvez-Myles R. USMLE Step 1 and 2 scores correlate with family medicine clinical and examination scores. Fam Med 2003;35:510-513.
Crawford III, CH, Nyland J, Roberts CS, Johnson JR. Relationship among United States Medical Licensing Step I, Orthopedic In-Training, Subjective Clinical Performance Evaluations, and American Board of Orthopedic Surgery Examination Scores: A 12-Year Review of an Orthopedic Surgery Residency Program. J Surg Edu 2010;67:71-78. doi: 10.1016/j.jsurg.2009.12.006.
Hemann, BA, Durning, SJ, Kelly, WF, Dong, T, Pangaro, LN. Referral for Competency Committee Review for Poor Performance on the Internal Medicine Clerkship Is Associated With Poor Performance in Internship. Mil Med 2015;180(suppl): 71-76. doi: 10.7205/MILMED-D-14-00575.
Casey PM, Palmer BA, Thompson GB, et al. Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments. BMC Med Educ. 2016;16:128. doi: 10.1186/s12909-016-0652-y.
Herndon JH, Allan BJ, Dyer G, Jawa A, Zurakowski D. Predictors of Success on the American Board of Orthopaedic Surgery Examination. Clin Orthop Relat Res 2009; 467: 2436–2445. doi: 10.1186/s12909-016-0652-y.
Johnson, GA, Bloom, JN, Szczotka-Flynn L, Zauner D, Tomsak RL. A Comparative Study of Resident Performance on Standardized Training Examinations and the American Board of Ophthalmology Written Examination. Ophthalmology 2010;117: 2435-2439. doi: 10.1016/j.ophtha.2010.03.056.
Julian ER. Validity of the Medical College Admission Test for predicting medical school performance. Acad Med. 2005;80:910-917.
Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: a meta-analysis of the published research. Acad Med 2007;82:100-106. DOI: 10.1097/01.ACM.0000249878.25186.b7

supplement4.docx

Download PDF

Journal Publication

published 10 Nov, 2020

Read the published version in BMC Medical Education →

Editorial decision: Major revision
25 Sep, 2020
Review #2 received at journal
21 Sep, 2020
Reviewer #2 agreed at journal
25 Aug, 2020
Reviewers invited by journal
26 Jun, 2020
Reviewer #1 agreed at journal
26 Jun, 2020
Review #1 received at journal
26 Jun, 2020
Submission checks completed at journal
16 Jun, 2020
Editor invited by journal
16 Jun, 2020
Editor assigned by journal
12 Jun, 2020
First submitted to journal
10 Jun, 2020

You are reading this older preprint version

Read the latest preprint version →

Identifying and Supporting Students at Risk of Failing the National Medical Licensure Examination in Japan Using a Predictive Pass Rate

Status:

Journal Publication

Version 1

Abstract

Background

Methods

Results

Conclusions

Background

Methods

Participants

Variables

Data analysis

Support To A Cohort Of Graduates In 2018

Results

Characteristics of those who failed and passed the NMLE in the 2012–2017 cohort

Logistic Regression Analysis

Ppr In The Nmle

Discussion

Limitations

Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Journal Publication

Version 1