The effectiveness of the IHC secondary school intervention was sustained for at least one year, but the effects were smaller. Like the results just after the intervention, the odds ratios for students were heterogeneous across the three trials, with the largest relative effects in Rwanda. Students in the control schools in Rwanda also had the lowest scores, as was the case just after the intervention. The adjusted differences for the primary outcome (passing scores) varied from 31.8% in Rwanda to 20.8% in Kenya. Just after the intervention, the adjusted differences for passing scores were 37.2% in Rwanda and 27.3% in Kenya.
The certainty of the evidence after one year is less for the main outcomes for students because of missing outcome data; 27% of the students enrolled in the trials did not complete the CTH test at one year. Loss to follow-up was similar in control and intervention schools. Likely reasons for their not taking the test include changing schools and dropping out. Older students and girls were slightly more likely not to have taken the test in both control and intervention schools. It is uncertain whether there were important differences between control and intervention schools in the reasons why participants were lost to follow-up that would potentially have biased the results.
The three trials included in this meta-analysis are the first large, randomized trials of a school-based intervention to teach adolescents to think critically about health interventions, and the only evaluation that has assessed long-term learning-retention (2, 3). A systematic review of school-based interventions to enhance adolescents’ ability to critically appraise health claims found one randomized trial and seven non-randomized studies (3). The authors concluded that “we know little about the effectiveness of educational interventions to teach adolescents critical appraisal skills.” A second systematic review of interventions to improve people’s understanding of key concepts for assessing the effects of health interventions found 14 randomized trials and 10 non-randomized studies (2). Five of those studies were in schools (grades 7–12) and all those studies were included in the first review. An updated search using the same search strategies used in the second review up to March 2023 (27) found one additional non-randomized study of a school-based intervention (grades 7–12) (28).
Process evaluations conducted alongside of the trials suggest that the teacher training workshop, completion of all the lessons, the design of the resources, the perceived value of the lessons, and administrative support contributed to the effectiveness of the intervention (Table 6) (29–31). Inadequate time, the fact that the IHC lessons were not included in the national curricula and examinations, and the lack of printed materials for students may have impeded the effectiveness of the intervention.
Table 8
Main findings of the process evaluations
| Factors & Potential impacts | Findings |
| Facilitators | |
| Teacher training workshop | Nearly all the teachers in intervention schools said the workshop was helpful or essential. They said it improved their understanding, motivation, and confidence to teach the lessons. |
| Delivery of the lessons | Teachers reported completing all the lessons and using the educational resources with minimal adaptation. They sometimes used the local language instead of English and used some local examples instead of the ones provided in the lessons. Most students attended most of the lessons. The teachers said that the students achieved the lesson goals, apart from the lessons on random allocation and random error. |
| Design of the resources | Teachers said the educational resources easy to access, understand, and use. They said most of the teaching strategies were familiar and easy to adapt, and a couple of teaching strategies were new to them and appreciated, such as use of response cards. Students said the lessons enjoyable and understandable. |
| Value of the lessons | Teachers, students, and other stakeholders all said the lessons were relevant to daily life and valuable. Teachers and students said this motivated them. They said the lessons addressed skills that are important for students, teachers, and the public. |
| Administrative support | Teachers in Kenya and Rwanda reported receiving support from their school’s administration, including time and resources to teach the lessons. |
| Barriers | |
| Inadequate time | Teachers were unable to complete the lessons in a single 40-minute period and generally used more time – sometimes as much as 120 minutes. Some teachers also reported not having enough time to prepare sufficiently due to competing demands on their time. The fact that it was an otherwise busy school term following school closures because of the Covid-19 pandemic may have contributed to this. |
| Curricula and examinations | Students, teachers, curriculum developers, and education authorities all identified the lessons not being in the curriculum or national examinations as a major barrier to implementing and scaling up the intervention. |
| Lack of printed material | Teachers, students, and some education authorities viewed the lack of a textbook or printed materials for students as a barrier to scaling up the intervention, since students lacked access to resources outside of the classroom. |
Effect modifiers
Overall, 47% of students in the intervention schools did not have a passing score after one year compared to 42% just after the intervention. This indicates that they did not have a basic understanding of the nine key concepts included in the lessons and would need additional or alternative instruction (11). Students in larger classes are less likely to have benefitted from the intervention, and students who performed poorly on end-of-term exams may be less likely to have benefitted. In addition, girls are probably less likely to have benefitted after one year.
It is uncertain why girls were less likely to have a passing score on the CTH test than boys. Gender inequality in educational outcomes differs between countries but persists in sub-Saharan Africa (32–34). A possible explanation for girls performing less well than boys is that in countries where women have a lower social status, education may be perceived as less important for girls (32). There also may have been biased instruction (33). Another possible explanation is that for many girls in low-income countries, adolescence is a time of extreme vulnerability, when they are under pressure from social norms and cultural practices that place restrictions on them (35).
Retention of what was learned
The effect of the IHC secondary school intervention on students’ and teachers’ ability to think critically about health choices persisted for at least one year. However, there was some decay in what was learned from the lessons. For students, there was about 12% decay in learning based on passing scores (adjusted for chance) and about 24% based on mastery scores. For teachers, there was about 4% and 13% decay based on passing and mastery scores respectively.
A large, randomized trial of the IHC primary school intervention also assessed outcomes after a year (36). The primary school children in that trial retained what they learned for at least one year. Their scores were, in fact, better after one year. The difference in learning retention between the primary and secondary school interventions could reflect differences in the learners, the contexts, or the interventions. The primary school intervention included textbooks that used a comic book story to explain and illustrate the key concepts and exercise books for the children. It included nine 80-minute lessons (about 12 hours altogether). Each lesson included review of the previous lesson, reading the textbook out loud, discussion, an activity, and exercises (37). The IHC secondary school intervention did not include resources that were handed out to the students. It included ten lessons designed to be taught in 40 minutes (about 7 hours altogether), and 40 minutes frequently was not enough time to teach the lessons. Possible reasons why learning retention was better for the primary school intervention are that there may have been more active learning and engagement of the children (38), and more practice testing (using the exercise books) (39).
A review of retention of basic science knowledge suggests that decay in what was learned in school is common, with only two-thirds to three-fourths of knowledge being retained after one year (40). To inform the choice of teaching strategies used in the IHC secondary school resources, we conducted an overview of systematic reviews of the effects of teaching strategies (41). Only 21 of the 326 included reviews reported learning retention as an outcome. Among the 37 strategies that we considered most relevant to teaching students to think critically about health, only two were found to improve learning retention. Practice testing is probably more effective than restudying for retention of knowledge and skills (39), and signaling to attract the learners’ attention and highlight important information probably improves retention (42).
We used teaching strategies that could improve retention. In each lesson, we included questions about the previous lesson, which the teacher asked the class. We suggested using response cards, which probably increase participation (43). We included a quiz in lessons 5 and 10, which were reviews of the previous lessons. The digital resources included printouts of the quizzes, but schools rarely printed materials for students due to the cost. We used signaling in the presentations included in the projector version of the lessons. We designed the lessons to include small group discussions, buzz groups (brief, intense discussions of a specific question with two or three people sitting next to each other), and class discussion to actively engage the students in learning. Teachers could decide which of these to use in each lesson. They often chose to use class discussion for several reasons. Small group discussion takes more time, both small group discussion and buzz groups can be difficult to monitor, and teachers were more familiar and comfortable with class discussions. The extent to which the teaching strategies used in the lessons affected retention of what was learned is uncertain since we do not have a reliable comparison.
Use of what was learned
The value of our intervention, or any other educational intervention, is limited if learners are unable to transfer what they learn to other contexts. There is uncertainty about how to achieve and evaluate transfer of learning (44). We assessed transfer or use of what was learned by students in the intervention schools based on self-report and observations by teachers in process evaluations, and using a ‘diary’, as reported in this meta-analysis.
Students and teachers that participated in the process evaluations reported using at least some of the key concepts they learned in their daily lives (29–31). Most of the IHC key concepts are applicable to many types of interventions unrelated to health, including environmental, educational, and social interventions (45). In the process evaluations, students also reported applying what they learned to decisions unrelated to health.
Based on self-report, most students in the intervention schools used what they learned from the lessons a little (37%) or a lot (51%). Most teachers in the intervention schools also noticed their students using what they learned from the lessons sometimes (63%) or a lot (25%).
When asked to identify and assess a claim that they had encountered during the past week, students in the intervention schools were more likely to assess the reliability of that claim correctly (adjusted difference 16%), but only 23% of students in the intervention schools correctly identified and assessed the reliability of a claim about the effects of health actions (Table 5). We did not find substantial differences in the reasons that students gave for choosing whether to take or recommend the health action, including consideration of the reliability of the claim, advantages of the health action, advantages of the health action, personal experience, and health professional or researcher advice.
Other studies have shown that critical thinking can be learned in ways that promote transfer to contexts outside of the classroom (46). However, available research evidence linking critical thinking to adult outcomes is limited and primarily based on associations (47). Nonetheless, research has found consistent positive associations of modest size between cognitive competencies and desirable educational, career, and health outcomes. Higher levels of educational attainment are associated with reductions in adverse health events and increases in healthy behaviors. The association between education and health behaviors might be due in part to greater trust of science and general cognitive skills, including critical thinking, which enable people to make better-informed health decisions (47, 48).
Adverse effects
Educational interventions can have undesirable as well as desirable effects, but potential adverse effects of educational interventions are rarely reported or even considered in evaluations of educational interventions (49–51). We assessed potential adverse effects of the IHC secondary school intervention by asking teachers to report any adverse events, qualitatively, and by asking students and teachers about potential adverse effects at one year. No adverse events were reported by teachers during the trial (7). Based on preliminary findings from the process evaluations and interviews with teachers (50, 52), we identified adverse outcomes that we assessed quantitatively in this study, including wasted time, conflict due to students challenging the beliefs or choices of others, decision-making harms due to misunderstandings, and stress caused by the lessons.
Because we only assessed these outcomes in intervention schools, we are unable to estimate the extent to which they are attributable to the intervention. For example, although there is some evidence of inequities in the extent to which students benefited from the IHC secondary school intervention, we were unable to estimate the extent to which the intervention causes inequities. Similarly, we were unable to estimate the extent to which the intervention caused participants to waste time and resources. Based on self-report, 10% of students reported that what they learned from the Be Smart about your Health lessons was less helpful than what they learned in other subjects (Table S9b). This suggests that for those students, the intervention may have caused them to waste their time, but we cannot estimate the size of this potential adverse effect in comparison to the standard curriculum. Moreover, 64% of students reported that what they learned was more helpful than what they learned in other subjects.
The IHC secondary school intervention could cause conflict between students and others by causing students to question other people’s claims, beliefs, or choices, and people becoming irritated or defensive. Most (84%) of the teachers in intervention schools reported that students challenged things that they said because of the Be Smart about your Health lessons, but only 10% said they experienced this negatively. Nonetheless, this suggests that the intervention may cause conflict and that consideration should be given to mitigating this; for example, by introducing and practicing non-confrontational strategies for questioning the basis for claims.
Another potential adverse effect of the IHC secondary school intervention is poor choices due to misunderstanding. A misunderstanding that we explored was that some students appeared to believe that unreliable claims that were used as examples in the lessons were reliable. Half (50%) of the teachers in intervention schools reported observing their students relying on unreliable claims that were used as examples in the lessons. We cannot quantify how many students did this or how often. Nonetheless, consideration should be given to mitigating misunderstandings of unreliable claims; for example, by ensuring clear communication about the reliability of the claims that are used, reiterating when a claim is unreliable, and reiterating that the focus of the lessons is on the key concepts, not on the examples that are used.
Some (18%) of the students reported that the Be Smart about your Health lessons were stressful or very stressful, but it is uncertain how this compares to other subjects or what caused the lessons to be stressful. For example, the lessons were an add-on to the standard curriculum and took time away from other subjects and preparing for examinations in those subjects. The extent to which this caused the lessons to be stressful is uncertain. Some teachers said preparing and teaching the lessons stressful. The most common reasons that they gave for this were that this took time away from other subjects and completing the lessons during the time available was stressful.
Limitations
A limitation of this meta-analysis is that the same research team was responsible for the meta-analysis, the included trials, the design of the intervention, and development of the outcome measure. To avoid bias in the assessments of the risk of bias and the certainty of the evidence, two researchers who were not involved in the trials made these assessments when the results just after the intervention were published (12). The same independent researchers assessed the risk of bias and certainty of the evidence for the one-year follow-up results. There was a moderate risk of bias for all the main outcomes because of loss to follow-up (Table S15), and lower certainty of the evidence (Table S16).
The questions we used to assess transfer, and the rubric used to code their responses were not validated. We pilot tested the questions and the overall results are congruent with the results of the CTH test. However, due to the burden of coding the answers to those questions, we only asked 10 students from each school to answer those questions. Consequently, there is insufficient data to reliably estimate the effects for each trial (country). In addition, although the data were coded independently by two people in each country, there may be differences between countries in how the coding was implemented.
This was the first time that teachers in the three trials taught the lessons, and the lessons were an add-on to the standard curriculum. Also, the lessons were taught in the first or second term when schools re-opened following closures due to the Covid-19 pandemic. This might have created additional stress for teachers and students. If the teachers had more experience, the lessons were in the curriculum, and there were normal circumstances, they might be more effective.
Nonetheless, the results after one year suggest that additional lessons are likely needed to reinforce what was learned, improve long-term learning-retention, and improve use of what was learned. It also is likely that more than seven hours of classroom time are needed, as well as additional lessons to ensure that all or nearly all the students benefit. Moreover, these lessons focused on just nine key concepts. Additional lessons are needed to teach other key concepts (6).
Implications
The results of this research show that adolescents in secondary schools in low-income countries with limited resources can learn valuable skills needed to decide what to believe or do in relation to health claims. This could help reduce susceptibility to misinformation about health interventions (53, 54). It also could help students to make better decisions about healthcare as they grow older, thereby reducing waste and unnecessary suffering (55). However, there are multiple other factors that limit the potential impacts on healthcare decisions and health, including a lack of access to health services and to reliable information (56). It is not easy to find evidence-based information, and there is a tremendous amount of misinformation (54, 57, 58). In low-income countries where many people have limited if any access to the Internet and to health professionals, accessing reliable information is especially challenging.
Inequities in the extent to which students benefit from the intervention might be reduced by translating the lessons for students lacking English reading proficiencies. Other strategies that might help include providing students with printed resources, more use of formative assessments and feedback, smaller classes, and teacher training focused on strategies for supporting students who need additional help (59). However, inequities in the outcomes of the IHC secondary school intervention likely reflect broader inequities in educational outcomes shaped by educational and other societal systems (32), and interventions to reduce educational inequities may need to start early in children’s lives (60).
Scaling up the intervention is likely to depend on incorporation of the IHC key concepts into the curriculum and examinations. The resources are freely available, are easily accessible using a web browser, and can be used offline. However, use of the resources in contexts outside of East Africa should be tested and may require translation and adaptation (61).
Additional lessons are needed to reinforce what was learned and to teach the ability to apply other key concepts. Human-centered design can help to ensure that resources for those lessons are experienced positively (4) and that they are effective. Ideally, children should start learning to think critically about health interventions and other interventions (45) as young as possible; and lessons should progress throughout their education, reinforcing concepts, skills, and dispositions that were learned, and introducing new ones (62).