Study setting and participants
We conducted a prospective randomized controlled blinded study at the University of Clermont-Ferrand, France, in February 2013 during the delivery of an ICU educational module (part of a postgraduate Intensive Care Unit (ICU) diploma program) attended by 25% of the national residents enrolled on the ICU training course. We enrolled all residents and lecturers who took part in the module into this study, except those who refused to participate. Written informed consent was obtained from all participants (residents and lecturers). The study protocol was approved by the Ethics Committee of the French Intensive Care Society (CE SRLF, No.12-394) and the local Institutional Review Board (IRB00008526, No.201837) in accordance with French law. The study was registered on the Clinical Trials Registry (clinicaltrials.gov) NCT01795651.
Study design and Intervention
The educational module was delivered over 5 consecutive days during which 12 expert lecturers each gave a single lecture pertaining to EBM for critical illness (Additional file 1). The residents were instructed to attend all lectures. They were sent the lecture topics 1 month before the module and therefore had the opportunity to prepare questions in advance. Interaction between the audience and lecturers was actively encouraged. All lecturers were experienced teachers who attended faculty workshops on effective lecture presentations, including how to devise an educational slide presentation and communicate THMs.
Two months before the educational module, the lecturers were randomly assigned in a 1:1 ratio to the intervention or control lectures group, by permuted-block randomization (i.e. random block sizes) using a computer-generated random allocation (Stata software). They were blinded to the study and their group assignment during the preparation and delivery of their lecture. The title and educational objectives of the lecture were chosen by the lecturer. The invitation e-mail provided information on the lecture conditions (face-to-face lecture with a slide presentation), the duration (30 min followed by 15 min for questions) and the learners’ characteristics (postgraduate resident doctors enrolled on an ICU training programme). At 1 month and at 1 week before the beginning of the module, a reminder e-mail was sent to each lecturer. The intervention element of the study was an explicit instruction to lecturers to include at least one slide entitled “Message” or “Take-home Message” into their slide presentations containing the written THM, for each THM delivered. A THM was defined as a short message of key relevant to medical practice. The number of THMs was limited to five per lecture. The choice and the wording of the THMs were decided on the lecturer. Each written THM was limited to 15 words. The instruction to the lecturers was clearly stated in a separate paragraph within the e-mail, which also included an article reporting the failure of residents to identify THMs in ICU postgraduate lectures [14]. The email did not mention that the lecture to be delivered was part of an experimental study. In the final part of the e-mails, the lecturers were encouraged to contact us for further details in the event of any problems or misunderstanding. The lecturers in the control group received only the three invitation e-mails, while those in the intervention group received the three e-mails including the instructional paragraph.
The lecturers were later informed about the study at the end of their lecture. All accepted to take part in the study. After the lecture, the lecturers provided the investigators with up to five THMs (≤ 15 words per THM) that they had included in their lecture, and a maximum of five multiple choice questions (MCQs) with answers related to their THMs. The lecturer indicated if their THMs had been explicitly written in the slide presentation. Two blinded teacher-reviewers validated this, for all 12 lectures. There was no disagreement between lecturers and reviewers.
The residents were informed about the study and gave consent to participate on the first day of the module, before the first lecture. However, they were blinded to the precise nature of the intervention, and to the outcomes, although they knew that they would be contacted 1 month after the final lecture to assess their knowledge of the module’s content. This assessment was not associated with other stake for the residents. One month after the last lecture, the residents completed an assessment form sent by e-mail. For each lecture, they were asked if they had attended the lecture, the THMs that they recollect (≤ 15 words per THM) and whether they had taken notes (defined as writing down the key points of the lecture). The definition of a THM was provided in the e-mail. The number of THMs delivered by the lecturer during each lecture was specified in assessment form. The residents did not have access to slide presentations but could consult their notes or learning materials when answering the assessment. They were allowed to give one additional THM to the number of THMs delivered during the lecture to increase the opportunity to recollect the THMs delivered by lecturer. Finally, they answered MCQs related to the different THMs. For each MCQ, the residents were given the number of correct answers. Two reminder e-mails were sent to residents who did not reply; if there was still no reply, they were secondarily excluded (Additional file 2).
Outcomes
The primary outcome was the difference in THMs retention as assessed by the rate of accordance between the THMs delivered by lecturers and given by residents, between the intervention and control lectures. The accordance was independently determined by two reviewers who were intensivist teachers but not lecturers on the educational module. They were also blinded to two group assignments. A binary scoring system was used: “Yes” when there was clearly a match between the resident and lecturer messages, and “No” to all other cases. If there was disagreement between the reviewers after a second analysis of THM accordance, a third reviewer analysed the data (disagreement arose in 4.6%, of the evaluation, n=174/3738). The order in which the THMs were listed on the responses forms was not taken into account in the analysis.
The second outcomes were 1) the difference in residents’ level of knowledge, as assessed by the MCQ, between the intervention and control lectures, 2) the identification of factors associated with better THMs retention or knowledge.
Each MCQ was rated 0 if there was at least one error among resident answers or 1 if there was no error. The knowledge of a resident was assessed for each lecture with a score based on the MCQs related to the lecture (total possible score, 100 points). Three groups of residents’ level of knowledge were established according to relevance and statistical distribution (interquartile range): low performance (<50 points), medium performance (50-80 points) and high performance (>80 points). If a resident failed to attend a lecture, no score was recorded.
Statistical analysis
At least 70 residents would participate in the educational module. Assuming that they would all attend all 12 lectures, 420 assessment forms were expected per lecture groups. In our previous observational study, the THM accordance rate observed was 39% at the end of lecture[14]. In the present study, the primary endpoint was assessed 1 month after the last lecture. Therefore, we assumed a 50% relative decrease in accordance, such that the expected THM accordance rate was 20%. For a two-sided type I error at 5%, 3,500 residents’ THMs (i.e. 1,750 THMs per lecture group) would have a power of 80% to show an absolute difference of 5% (20% vs. 25%), taking into account between- and within- resident’s variability measured using intra-cluster correlation coefficient (ICC).
All statistical analyses were performed with Stata statistical software (version 13, StataCorp, College Station, US). Categorical data are expressed as numbers and percentages, and quantitative parameters as mean ± standard-deviation or median [interquartile range], according to statistical distribution. The normality of the data was assessed using the Shapiro-Wilk’s test. To take into account variability between and within lecturers and residents, random-effect models were generated (lecturers and residents as crossed random effects). These models (generalized linear mixed model with logit link function) were used to determine factors associated with THMs retention and residents’ knowledge. Multivariable analyses were then performed, with covariates (fixed effects) determined according to their significance in univariate analysis (P<0.10) and clinical relevance, for THMs retention: gender, slides per lecture, notetaking and THMs written on slides; for residents’ knowledge: gender, notetaking and THMs retention. Particular attention was paid to multicollinearity and the interactions between covariates, and the impact of adding variables to, or omitting them from, the multivariable model. Results were expressed as odds-ratios (OR) or adjusted odds-ratios (aOR) for multivariable analyses and 95% confidence intervals (95% CI). Sensitivity analysis was conducted with THMs retention rate treated as a continuous variable (using negative binomial generalized linear mixed model) and categorized according to various cut-offs, such as a cut-off of 25% determined according to the expected assumption used for sample size estimation.