Correlation of Residents’ Performance in Competency-Based Exams and Orthopaedic In- Training Examinations (OITE)

doi:10.21203/rs.3.rs-426917/v1

Download PDF

Research Article

Correlation of Residents’ Performance in Competency-Based Exams and Orthopaedic In- Training Examinations (OITE)

https://doi.org/10.21203/rs.3.rs-426917/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: The research team aimed to assess the relationship between performance in a competency-based curriculum (CBC) evaluation, and the Orthopaedic In-Training Examination (OITE) in the Postgraduate Year 1 (PGY1) cohort of 2016-2017.

Methods: After development of the ‘Basic Trauma’ (BHT) and ‘Basic Arthroplasty’ (BA) CBC modules, assessment consisted of multiple-choice questions (MCQ), objective structured clinical evaluation (OSCE), structured oral panels, and the OITE were conducted annually. We collated MCQ and OSCE evaluations for BHT and BA, as well as the OITE result for the same cohort from PGY1 and the end of PGY2. We evaluated the OITE score difference for correlation with the scores attained for the two CBC modules.

Results: Among all participants (n=9), there was a significant improvement in mean OITE scores from PGY1 to PGY2 (43.78% (±4.09) to 56.67% (±4.24); t-test p= 0.00). There was no significant correlation between OITE improvement, and scores attained in the BHT and BA modules, nor between BHT and BA results and the ‘Hip and Knee’ and ‘Trauma’ domains of the OITE exam.

Conclusions: Improvement in OITE performance is not dependent on evaluated CBC modules. Further research to determine what factors play a role in trainee improvement in objective performance is required.

Educational Philosophy and Theory

Internal Medicine

competency-based medical education

orthopaedics

Orthopaedic In-Training Examinations

Canada

The competency-based curriculum (CBC) has been adopted as the sole mode of training and assessment in orthopedic residency training at the University of Toronto since the 2013/14 academic year.¹ The CBC was designed to not only meet the current accreditation and educational requirements of the postgraduate medical education accreditation body in Canada, the Royal College of Physicians and Surgeons of Canada (RCPSC), but also in anticipation of meeting the requirements for an initiative in Canada to transition all training programs to competency based medical education (CBME).¹ The curriculum is divided into 21 modules spread over the 5 years of residency training, as shown in Fig. 1. The seven CanMEDs roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) were embedded within the modules. Figure 1 identifies in abbreviated form the specific CanMEDS taught and assessed in each module.

The CBC was adopted after an initial pilot carried out by the University of Toronto in 2009.² The most significant change that the CBC pilot design brought to residency training was that trainees would be allowed to progress to the next learning module only if the objectives of the module they were currently on were achieved,³ ensuring that the surgical training time was not time-based.¹

Developed in 1961 as the first test of its kind, the Orthopedic In-Training Examination (OITE) is a 275-question multiple-choice examination produced by the American Academy of Orthopedic Surgeons (AAOS) and is administered annually to more than 4300 orthopedic surgery residents worldwide.^4,5 The OITE is the only standardized examination prior to written board examinations that residency programs can use to track resident progress and identify early problems.⁵ OITE scores have been shown to correlate with residency performance,⁶ with the exam itself being noted to be more important to program directors in the United States than it is in Canada.⁷ Program directors of residency programs in Canada focus on other means of evaluation apart from the OITE, with the CBC curriculum developed by the University of Toronto being an example of this. These methods include Objective structured clinical examinations (OSCES), internal examinations, In-Training Evaluation Reports (ITERs) versus the OITE in assessing residents’ ability in the early years of training.

Two of the core components of the CBC curriculum in the 1st year of residency training at the University of Training are the ‘Basic Trauma’ and ‘Basic Arthroplasty’ modules. With this study, we aimed to assess the relationship between trainee performance within these components of the CBC program and results attained in OITE in the Postgraduate Year 1 (PGY1) cohort of 2016–2017.

We developed a curriculum map, evaluations and several content reviews for the Basic Trauma and Basic Arthroplasty CBC modules for the PGY1 cohort in 2016–2017. Each evaluation consisted of multiple-choice questions (MCQ) and an objective structured clinical evaluation (OSCE). We established an evaluation center with faculty & fellows as evaluators. Nine residents completed each exam panel biannually after successful completion of the respective modules. Residents received real-time feedback upon completion of the MCQ and structured oral panels.

We collated the residents’ scores (average percentage score from multiple choice and OSCE assessments) on 2 defined competency-based curriculum (CBC) modules (‘Basic Arthroplasty’ and ‘Basic Trauma’). In addition, each resident completed the Orthopaedic In- Training Examination (OITE) annually. We collated the overall OITE score as well as the scores for the ‘Hip and Knee’ and ‘Trauma’ domains in OITE. We used Pearson’s correlation coefficient to assess if the difference in the overall OITE score over the 2-year period was associated with the scores attained in the ‘Basic Arthroplasty’ and ‘Basic Trauma’ modules. We used students’ t-test to assess for differences in the mean examination scores attained, with a p value < 0.05 deemed statistically significant.

We obtained REB approval for educational research pertaining to the CBC program of orthopaedic surgery.

Data was available for nine residents. There was an improvement in mean OITE scores for the whole cohort from PGY1 to postgraduate year 2 (PGY2); 43.78% (±4.09) to 56.67% (±4.24), which was statistically significant (t-test p = 0.00). There was no significant correlation between the improvement in OITE scores and the scores attained in the ‘Basic Arthroplasty’ and ‘Basic Trauma’ modules (Pearson correlation R values − 0.032 (p = 0.936) and 0.103 (p = 0.792), respectively). No significant correlation also existed between the CBC module performance and the scores attained in the ‘Hip and Knee’ and ‘Trauma’ domains of the OITE exam. There was a positive correlation between the scores achieved for Basic Trauma and Basic Arthroplasty modules (Pearson correlation R value 0.873 (p = 0.002)). Table 1 demonstrates the mean OITE and CBC module scores for the cohort whilst Table 2 illustrates the results of the correlation analyses.

Table 1

OITE and CBC modular scores for PGY1 residents 2016/2017.
Resident ID	OITE Score 2016 (%)	OITE Score 2017 (%)	Improvement in OITE Score (% Change)	Basic Trauma Score (%)	Basic Arthroplasty Score (%)
1	51	62	11	75.00	73.00
2	38	55	17	84.95	80.00
3	41	51	10	81.6	77.00
4	41	60	19	70.00	68.35
5	46	62	16	72.50	76.19
6	48	51	3	78.75	74.23
7	43	54	11	76.79	73.65
8	41	58	17	90.00	82.00
9	45	57	12	90.6	79.05
Mean	43.78	55.67	12.89	76.88	78.00
Standard Deviation	4.09	4.24	4.94	8,98	3.92

Table 2

Correlations between change in OITE score and CBC modular assessments for PGY1 residents (2016–2017).
	OITE Difference Overall	OITE Difference Trauma	OITE Difference Hip Knee	Trauma Score Mean	Arthroplasty Score Mean
OITE Difference Pearson Correlation Sig. (2-tailed) N	1 9	.612 .080 9	.408 .275 9	− .032 .936 9	.103 .792 9
OITE Difference Trauma Pearson Correlation Sig. (2-tailed) N	.612 .080 9	1 9	.072 .855 9	− .109 .780 9	.111 .776 9
OITE Difference Hip Knee Pearson Correlation Sig. (2-tailed) N	.408 .275 9	.072 .855 9	1 9	− .273 .477 9	.061 .876 9
Trauma Score Mean Pearson Correlation Sig. (2-tailed) N	− .032 .936 9	− .109 .780 9	− .273 .477 9	1 9	.873** .002 9
Arthroplasty Score Mean Pearson Correlation Sig. (2-tailed) N	.103 .792 9	.111 .776 9	0.61 .876 9	.873** .002 9	1 9

There are several factors that predict higher overall OITE scores in the literature, which include 1) higher PGY level, 2) program location in the United States, 3) having consequences in the program for residents who achieve low scores, 4) emphasis on the OITE to determine study content in the last year; and 5) greater hours per week spent reading general orthopedics in the month prior to the most recent OITE.⁷ There is also evidence that lower United States Medical Licensing Examination (USMLE) scores along with consistently low OITE scores likely identify residents at risk of failing future written board examinations.⁵ The duty hours worked by residents’ has also been found not to correlate with the OITE scores attained.⁸

To our knowledge, no previous study has attempted to compare the relative efficacy of using alternative methods of evaluation such as the CBC modules described in this study with the OITE. Although the OITE score improves from 2016 to 2017 in this cohort of PGY1 residents, this improvement cannot be attributed to participation in the CBC modules, particularly as no significant relationship exists between the module scores, and the domains of trauma and arthroplasty within the OITE examination.

There are several factors that can potentially contribute to the general improvement in the OITE score from 2016 to 2017, which include increasing clinical experience, improvement in utilization of learning resources, and perhaps better self-directed learning. Motivation, good study habits, and work ethic lead to consistently strong performances on standardized testing such as OITE.⁵

Limitations of this study include the small number of residents evaluated as well as the use of only two components of the CBC curriculum. We believe that the comprehensive nature of the evaluation process for the two modules identified may provide insight into which predictive factors could possibly influence OITE performance.

In conclusion, improvement in OITE exam performance in this cohort does not appear to be dependent on assessed performance in the evaluated CBC modules. Further research is required to understand what other factors may play a significant role in trainee improvement in competency-based evaluation, and objective performance.

CBC

Competency-based curriculum

OITE

Orthopaedic In-Training Examination

PGY1

Postgraduate Year 1

BHT

Basic Trauma

Basic Arthroplasty

MCQ

Multiple-choice questions

OSCE

Objective structured clinical evaluation

PGY2

Postgraduate Year 2

AAOS

American Academy of Orthopedic Surgeons

ITERs

In-Training Evaluation Reports

USMLE

United States Medical Licensing Examination

Ethical approval and consent to participate

Ethical approval was granted for this study was obtained from the University of Toronto Ethics Review Committee in June 2007. Participation was voluntary and informed consent was obtained from all participants prior to data collection. All methods were carried out in accordance with relevant guidelines and regulations

Consent for publication

All subjects involved in all methods have provided consent for participation and publication, whether written for the randomized controlled trial or verbal for the workshops and focus groups.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on request.

Competing interests

Not applicable.

Funding

Not applicable.

Author’s contributions

TO, PM, JH, PF, MN, JW, HK, TA, TD, OS, WK, ST, FM, PW, SW, SB, TM, YZ, EG, MC, HF, JM, JM, GS, MG, CE, PW, CW, IW, JP and VW made substantial contributions to the study’s conception and design, data collection, analysis and interpretation. TS provided critical input in writing and preparing the manuscript for submission. All authors contributed to drafting, editing and revising the manuscript. All authors have read and approved the final manuscript.

Nousiainen MT, Mironova P, Hynes M et al. Eight-year outcomes of a competency-based residency training program in orthopedic surgery. Med Teach. 2018; 18: 1–13. https://www.ncbi.nlm.nih.gov/pubmed/29343150
Alman BA, Ferguson P, Kraemer W, Nousiainen MT, Reznick RK. Competency-based education: a new model for teaching orthopaedics. Instr Course Lect. 2013; 62: 565–569. https://www.ncbi.nlm.nih.gov/pubmed/23395058
Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from flexner to competencies. Acad Med. 2002; 77: 361–367. https://www.ncbi.nlm.nih.gov/pubmed/12010689
Marsh JL, Hruska L, Mevis H. An electronic orthopaedic in-training examination.J Am Acad Orthop Surg. 2010; 18(10): 589–596. https://www.ncbi.nlm.nih.gov/pubmed/20889948
Dyrstad BW, Pope D, Milbrandt JC, Beck RT, Weinhoeft AL, Idusuyi OB. Predictive measures of a resident's performance on written Orthopaedic Board scores. Iowa Orthop J. 2011; 31:238–43. https://www.ncbi.nlm.nih.gov/pubmed/22096449
Spitzer AB, Gage MJ, Looze CA, Walsh M, Zuckerman JD, Egol KA. Factors associated with successful performance in an orthopaedic surgery residency. J Bone Joint Surg Am. 2009; 91(11): 2750–5. https://www.ncbi.nlm.nih.gov/pubmed/19884456
Evaniew N, Holt G, Kreuger S, et al. The orthopaedic in-training examination: perspectives of program directors and residents from the United States and Canada. J Surg Educ. 2013; 70(4): 528–36. https://www.ncbi.nlm.nih.gov/pubmed/23725942
Bohm KC, Hill BW, Braman JP, Ly TV, Van Heest AE. Orthopedic Residency: Are Duty Hours Predictive of Performance? J Surg Educ. 2016; 73(2): 281–5. https://www.ncbi.nlm.nih.gov/pubmed/26774934

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Correlation of Residents’ Performance in Competency-Based Exams and Orthopaedic In- Training Examinations (OITE)

Status:

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1