Assignment of the students to instructional approaches of the skills
The assignment of students to one the study arms occurred prior to the training week, independent of the authors and independent of study participation by the deanery. The allocation of the learning group in the study to the three instructional approaches was performed alternately.
Study design
All ethical principles for medical research involving human subjects dictated by the 1975 declaration of Helsinki, as revised in 2013, were considered. Concordant with the Ethics Board at the University Medical School, no ethical permission was necessary for conducting the study. The study was prospective, blinded, randomized, and controlled with the following parallel arms (feedback methods):
Control group: Direct expert feedback (DEF) (n = 20)
Intervention group 1: Individualized video feedback (IVF) (n = 20)
Intervention group 2: Unsupervised video feedback (UVF) (n = 20)
Study participants were sixty fourth-year dentistry students from the University Goethe of Frankfurt in the period of 2018-2019 attending a compulsory internship, which includes a five-day rotation through every section of the Department of Oral, Cranio-Maxillofacial and Facial Plastic Surgery, i.e. the operative room, the outpatient clinic or the emergency department. Before starting their rotation, students have to complete a practical skills training. The aim is to give dentistry students a short overview of the most common consultation reasons in OMS and prepare them for the upcoming clinic rotation. It is divided into a theoretical part (240 minutes) in the morning and a practical skills training (240 minutes) in the afternoon in which the study took place. Trained practical skills include performing a structured facial examination, placing a venous catheter and an interdental ‘Ernst’ ligature. Lessons were held in small groups ranging from four to six students. 14 Sixty students signed an informed consent of participation after receiving explanation of the study process and objectives, from which they could withdraw at any time (Figure 1). Evaluation of performance was done using previously validated global rating scales (Figure 2 and 3).15
The study measured the student’s improvement of performance and time used to perform a task directly after an introduction exercise (T0) and a second exercise after receiving feedback (T1). Finally, to measure the long-term learning retention, a final examination (T2) was performed six weeks after the post-test. Examiners were blinded to the study group and received an educational course as calibration and to gain experience using the global rating scale. 15
Pre-test evaluation
Practical skills training and measurements of pre-test evaluation
In the practical skills training, an emergency situation was simulated in which the students had to insert a periphery venous catheter. In the first exercise, it was evaluated the correct use of gloves, placement of a tourniquet, knowledge of periphery-venous anatomy, preparing a sterile working surface, placing the catheter and fixation of the catheter (Figure 2). The second exercise involved the first aid treatment of a mandibular fracture using an interdental ligature technique (‘Ernst’ ligature). In this exercise, the correct identification of the fracture line, placement of the ligature, cutting and twisting the endings of the wire and checking the stability of the ligature were evaluated (Figure 3). Before taking part in the practical skills training, all students received instructions through a standardized teaching video of each skill. Afterwards, students performed the skills by themselves for 30 minutes. After this, students were video recorded performing each skill as a performance measurement prior to receiving one of the feedback methods investigated in this study. Additionally, time of execution of the practical skills performance was documented.
Feedback Methods
Direct expert feedback (Control group)
In this group, after performing each skill by themselves, every student was video recorded for the later blinded performance assessment while performing the assigned tasks once. Immediately after this, the students practiced each skill again for 30 minutes while receiving a direct feedback regarding their performance by an instructor (T0). Feedback was given using a five-step feedback sheet. These five steps in the feedback protocol assessed what went well, what could be improved, what went badly, what was missing, and what was the take-home message for each student. Immediately after the feedback, the students practiced again for 30 minutes before repeating the exercise while again being video recorded for subsequent assessment (T1).
Individualized video feedback
In this group, after performing each skill by themselves, every student was video recorded for the later blinded performance assessment while performing the assigned tasks once. This initial video was then reviewed by the student and an instructor. Based on the video, feedback was given by the instructor using the same five step-feedback sheet (T0). The feedback sessions were held in a group setting and lasted for 30 minutes. Immediately after the feedback, the students practiced again for 30 minutes before repeating the exercise while again being video recorded for later assessment (T1).
Unsupervised video feedback
In this group, after performing each skill by themselves, every student was video recorded for the later blinded performance assessment while performing the assigned tasks once. Students then reviewed their own performance. As feedback, students received once again the standardized video instructions and were handed out the same five step-feedback sheets (T0). The feedback sessions were held in a group setting and lasted for 30 minutes. Immediately after the feedback, the students practiced again for 30 minutes before repeating the exercise while again being video recorded for later assessment (T1).
Performance measurement
To assess the acquired competence in both skills of the study, standardized checklists were used during the practical skills training week before (T0), directly after the intervention (T1) and 5–13 (SD = 3.16) weeks later (T2), as part of the curricular and summative OMF OSCE (eight stations in total). A trinary scoring scale was used (zero points for not done, 1 point for done but incorrect, and 2 points for done and correct) for each checklist, which was based on the checklist used in the tutor manual (Figure 2 and 3).1516
During their practical skills training, students were video recorded (Camera System: Panasonic HC-X929, Osaka, Japan) for later performance measurement and to validate the used checklists by two independent, blinded examiners. All examiners were blinded toward the students’ instructional approach and affiliation of the learning group. They received training before the OSCE to gain experience in the use of the checklist.
Statistical analysis
Microsoft Office 2016 (Microsoft Office 2007, © Microsoft Corporation, Redmond, USA) for Mac and SPSS Statistics version 19 (IBM, Armonk, USA) were used for the statistical analysis. All the data collected were analyzed using two-way analysis of variance (ANOVA) with a Tukey multiple comparisons test (α = 0.05, 95% CI of diff.) of all pairs. Since the sample size was small, Cohen’s d was used as an additional control test to support the interpretation of the data. Cohen’s d is defined as the difference between two means divided by a standard deviation for the data, resulting in a unitless value that helps to interpret the effect size of observed results and, hence, the statistical power of a study. For most types of effect sizes, a larger absolute value indicates a stronger effect. Furthermore, it can be used as an additional control test since prior studies have shown that significant test results alone are not sufficient to interpret data and draw conclusions. 17 The results are presented as the mean and standard deviation (SD) and depicted in tables. Statistical significance was considered if p <0.05.
Sample size estimation
Based on prior examination results from the years before the intervention, we estimated an average student performance of 70% with a standard deviation of 10% in the OSCE. With an average student number of 65 per semester, a size of 56 was calculated based on the following parameters: average student performance = 70%, alpha = 80%, beta = 20%.