This is the first study evaluating the impact of formative OSCEs on summative OSCE success in an EM training programme. All residents who participated in more than three formative OSCE succeeded in the summative OSCE on their first attempt, suggesting a positive training effect where the cumulative experience gained from participating in multiple formative OSCEs was beneficial to residents taking the summative OSCE. This is supported by the upward trend in residents’ scores over successive formative OSCEs.
The qualitative responses provide an explanation for this finding, with over half of residents surveyed stating that the formative OSCE was useful in helping them to familiarise themselves with the examination format. This corroborates with previous studies [19, 23]. It has been found that unfamiliarity with the format or tested clinical competencies of an OSCE is a major factor that limits performance in the OSCE, and that learners show more consistent performance levels after taking multiple formative OSCEs [16]. In our study, repeated practice and evaluation in multiple formative OSCEs reduced the surprise element of the examination and helped to improve residents’ confidence and performance. However, attending one formative OSCE was not associated with first-attempt OSCE success, suggesting that multiple formative OSCEs were required to provide adequate exposure.
A second benefit of taking multiple formative OSCEs was the opportunity for residents to practise their examination skills in a safe training examination, as expressed by two in five residents. Formative examinations are perhaps particularly valuable in preparing for OSCEs which, unlike written examinations, test not only learners’ factual knowledge but also their abilities to interact effectively with real and simulated patients, and to synthesise and communicate information succinctly to examiners [4]. Moreover, EM OSCEs tend to place a high importance on resuscitation and procedural skills, and there is evidence that these skills are more effectively learnt through simulation than traditional forms of education [24]. This may be because these stations assess leadership, prioritisation and time-sensitive skills, and require specialised resources. They are therefore more difficult for residents to practise on their own.
In the programme, junior residents are allowed to choose when to take the summative OSCE. It is therefore useful to have an indicator to guide residents on whether they are ready to attempt the summative OSCE. Several residents felt that the formative OSCE was useful for assessing their progress and gauging whether they were ready to take the summative OSCE. This was supported by the tendency of residents who succeeded in the summative OSCE to have higher mean and proximate formative OSCE scores. Although these trends were not statistically significant, this was likely due to the small proportion of residents (5/40 residents, 12.5%) who failed the summative OSCE. This low failure rate may be partially due to residents only attempting the summative OSCE when they felt adequately prepared for it, with one of the contributing factors being their formative OSCE performance. Hence it is of practical significance, including the cost savings of the summative examination fees in the event of failure, that the formative OSCE could provide feedback to residents about whether they were ready to take the summative OSCE, and help faculty identify outstanding residents and weaker-performing residents who required more educational intervention.
Residents commented that the formative OSCE enabled them to identify their strengths and weaknesses. Though primarily a practice exam that acts as an “assessment of learning”, the formative OSCE was also an “assessment for learning” [25]. Previous investigations have found that testing enhances long-term retention of knowledge more effectively than repeated studying [26, 27]. By requiring active engagement from residents [28], the formative OSCE stimulated self-reflection and encouraged further studying. In addition, the incorporation of a feedback component allowed faculty members to highlight areas for improvement and give constructive advice, with residents commenting that the feedback was helpful in identifying deficits in their knowledge and skills and allowing them to learn from their peers’ mistakes. Previous studies have found that even short periods of immediate feedback can improve learners’ competencies [29].
While the formative OSCE was universally considered useful, a few residents raised that the formative OSCE was of limited benefit to those who did not prepare beforehand for them. The OSCE tests “demonstration of learning” on Miller’s Pyramid. Learners therefore require an adequate level of prior theoretical knowledge to reap maximal benefits from an OSCE. Hence the formative OSCE may be of limited benefit to residents who are not contemplating taking the summative examination yet and have not begun to prepare.
There were some limitations. The study was performed in a single residency programme focusing on the MMed and MRCEM examinations, and further research is needed to evaluate if the results are applicable in other training programmes or other examinations. Only a small number of residents did not succeed in the summative OSCE on their first attempt, and while this may be an encouragement for the programme, it limited our ability to assess factors that contributed to summative OSCE success. A control arm did not exist as most residents participated in the formative OSCE prior to attempting the summative OSCE, limiting our ability to compare the results of residents who did and did not participate in formative OSCEs.