MindPower is an adjusted and upscaled version of the CWD/CWS interventions. CWD/CWS courses have traditionally been delivered within a health service context by health personnel targeting high risk groups to prevent depression. In contrast, MindPower is delivered class wise, in the school, during ordinary school hours, by trained school teachers, universally, i.e. independent of the students’ risk of mental distress or disorder, to strengthen positive mental health and quality of life.
In this study we tested effects on self-efficacy, self-esteem and quality of life by using a two-groups cluster randomized delayed intervention design with linear mixed models statistical analyses. The intervention was offered to all students at all schools in a county in South-East Norway. Level of self-efficacy among the participants was compared to a large scale population survey (UngData), nationwide and from the same schools as were the MindPower-intervention was implemented.
The main finding was that there was no significant effect of the MindPower program delivered universally by trained teachers in a class room situation. Neither did we find effects within any of the two intervention groups, nor did the two groups change consistently as expected from the delayed intervention design if an intervention effect had occurred. These were consistent findings across outcome measures, except a small significant decrease in quality of life.
Another finding was that the level of self-efficacy in both our intervention groups was high and equal both to that of the general population in the same age groups, nationally and in the same schools as were the intervention was implemented. This indicates that the students that participated in the intervention were characterized by high levels of self-efficacy, self-esteem, and quality of life from the beginning to the end of the study. A further increase in these measures through a universal initiative would therefore be a challenging task (ceiling effect).
The high levels of self-efficacy, self-esteem and quality of life uncovered in UngData in combination with very high and steadily increasing levels of self-reported mental distress as well as symptoms of depressive feelings and anxiousness found in Scandinavian studies may seem contradicting. Particularly if the increases in these self-reported negative feelings are interpreted as indications of a deterioration of the mental health among young people.
However, if the increasing reports of negative feelings are regarded as a sign of de-stigmatization and increased openness, such an effect is compatible with the high levels of self-efficacy, self-esteem and quality of life observed in this study. That would be highly consistent with the very high levels of satisfaction with life reported among close to 90 percent of, at least Norwegian, young people (UngData).
In that case, a displacement of the meanings of values in scales intended to measure mental distress and symptoms of common mental disorders is likely to have occurred. Because the context of openness has changed, items like the Hopkins Symptom Checklist (SCL-8) for symptoms of depression and anxiety, does not mean the same today as it meant a decade or two ago.
This may explain why we with such a carefully and comprehensively prepared intervention, advanced study design and statistical methods, did not find any effects of MindPower on self-efficacy, self-esteem or quality of life in this study. We may already have hit the ceiling before we started.
Another explanation could have been selective drop out as the drop out from this study was progressive and comprehensive. However, the complete responders on the last assessment were not significantly different from those who dropped out. A clinically meaningful effect size of the intervention should therefore have been revealed in spite of the high level of drop out.
A third possible explanation may be lack of fidelity. Our monitoring of fidelity indicates that only half of the students in both intervention groups reported having been tuitioned at their tenth and last Mindpower session. This deviates from the instructions in the course leader manual and the study design. Completion of all the ten sessions are regarded as imperative. Less than complete implementation may attenuate effects of the intervention seriously.
What can we learn from this study?
The planning of this study was careful and comprehensive. The study was launched and conducted in a very positive contextual atmosphere. The government had pledged a change in the curriculum. A new national curriculum on public health and life skills, including mental health, was in the process of being launched in all schools in the country.
Parents and students themselves, teachers, researchers, psychologist and local, regional, and national politicians had for a long time argued for equalization of mental and physical health in the school.
The top administrator of education in the county welcomed the study and was effective in supporting communication and collaboration with the school administration and the schools. There were several motivation meetings with the teachers before and during the project period indicated high motivation to participate among students, school leaders and teachers.
There were comprehensive preparations before the implementation. The organization “Fagakademiet” arranged intensive training of all the involved teachers, e.g theoretical lectures, practical guidelines and instructions, role playing, and homework. According to an internal evaluation from Fagakademiet, teachers reported very high satisfaction regarding the training.
Teachers received a short list of key points prior each data collection, to ease remembering of what to do. All 170 teachers and the ten principals received the study-coordinator’s phone number and e-mail address in case they had questions about the study. There was updates and information by e-mail and by phone, and meetings at each school.
With such a positive context and such a thorough implementation and enthusiastic follow up, how could it go so wrong?
A qualitative study connected to this trial has addressed this question (Cheng, Ihlebæk & Sælid, 2021). Furthermore, we have collected information through a number of informal sources. This has provided us with a number of learning points for others who intend to launch large-scaled universal mental health initiatives among public high school students in a similar context.
The core of a successful evaluation is to know to which extent essential factors have been implemented. In this study, however, a full-scale fidelity on teachers and school administrations study was not possible.
It is crucial that teachers follow the instructions in the course leader manual on how to tuition the intervention program. Also, follow the instructions in the study design regarding which course day are to be tuitioned.
In implementing a study like this, the first point is to monitor the extent to which the teachers start the intervention at the correct time point. In this study, half of the MindPower sample was instructed to start four months prior to the second group. However, without a full-scale implementation study of the teachers, optimal assurance that the expected starting point is realized exactly, is not possible to achieve.
Another implementation challenge is that at some schools not all teachers may have sufficient support from their school administration. Teachers may also lack assistance from their colleagues in adapting the MindPower program into the curriculum. As a consequence, some classes may never have started the intervention, while others may have received a shortened program.
Here, one solution may be, as some schools did, to establish a forum at the school that could include mental health personnel. Their intention was to share positive and negative experiences and to seek advice and support. Such a team might give the teachers the support they need to find time to MindPower in the curriculum. Additionally, this may contribute to feelings of competence in teaching life-skills.
Still another challenge is to what extent the teachers experience to have a clear job description. Some teachers might feel that teaching life-skills is providing therapy, a task which is not a school teacher’s responsibility. This may result in lack of motivation and avoidance of teaching mental health-related programs.
One solution to this may be that the program leadership and principals together give counseling on role consciousness and give clear directions in these concrete cases. Life-skills training is not therapy and can be taught in terms of “how life is”, implying that challenges in life are normal and that no one is alone in experiencing such feelings.
In our study, some teachers were instructed to participate in the preparatory course without being asked about their motivation. Some school administrators did not allow teachers to choose not to teach MindPower. Especially some teachers who taught practical vocational subjects such as electronics, plumbing, and carpentry were not comfortable carrying out teaching activities implying handling of issues related to emotions.
To address this, one solution is to only admit motivated teachers to teach life-skills programs. If the teachers are not motivated, they will not do a good job. It is also necessary to respect that some teachers feel uncomfortable in teaching life-skills. Consequently, life-skills teaching or training should not be mandatory.
Furthermore, a challenge may be to what extent the MindPower program has been sufficiently well designed and tailored for classroom-based educational purposes. In our study, some teachers experienced difficulties running the MindPower program in the classroom.
A solution to this might be to involve teachers and students in the planning of the implementation. Ideas and opinions of teachers and students are crucial in order to succeed. In this way the program may be better tailored for the particular school-setting. In our study, students and teachers were asked to give feedback prior to the implementation. Based on these feedbacks, the designer of MindPower changed certain points of the program to ease the implementation process, e.g. 90 minutes sessions rather than two and a half hours as in traditional Coping With Depression/Coping with Strain-courses (Cuipers et al., 2009; Sælid et al., 2016).
We experienced that teachers had deviated from the implementation plan on several points. For instance, some teachers did not complete all the ten MindPower sessions. Other teachers shortened the 90 minutes sessions. This emphasizes the necessity do involve all teachers on beforehand and make sure that the school administration expects the program to be implemented as planned and do not accept deviations from the plan. A more flexible approach, however, would be to work on the program to gain an acceptable format for all participants.
The confounding variables and the implementation factors in large studies are not easy to control. The current study has an RCT-design. However, researchers should keep in mind that the findings from smaller, well implemented studies with sufficient statistical power, may be more valid than a large scaled natural study were several confounding factors may be out of control (Dawson, Yeomans & Brown, 2018). In our case, a smaller study might have done it easier to have better control of some of the implementation factors described above.
One of the greater learning points from this study concerns technicalities in collecting data from public high school students. The data collection department at NIPH, a governmental research institution, refused to use the student e-mail addresses provided by the students’ teachers. Instead, they insisted on collecting and using the students’ officially registered e-mail addresses.
However, this register contained to a large extent the students’ parents’ e-mail addresses and were not valid contact points to the students. Furthermore, some e-mail addresses received from the teachers were also not valid. Consequently, we ran into difficulties in administering the questionnaires to some students’ valid e-mail addresses. These addresses a common problem. Young people may frequently use unofficial e-mail addresses and change e-mail addresses frequently. Washing addresses against official registers may do more harm than good. In addition, at each data collection time point, some students had difficulties in both retrieving and opening the questionnaires.
If a technical data collection partner is involved, which administrates the questionnaire, a contract is advised where the data collection partner fully accepts the plan for what to do if there are technical failures. Such a plan could include setting up additional links with the questionnaires, which can be sent to the participants and teachers if technical problems occur.
Pretesting of the data collection procedures is warranted. This should include making sure that you have the best possible records of e-mail addresses and telephone numbers of all study participants. Finally, if possible, change technical partner if the problems continue, e.g. the questionnaires are not possible to be opened for all students at any time point.
Altogether, these challenges resulted in high drop-out rates and potential sample bias. In our case, fortunately, the complete responders on the last assessment were not significantly different from those who dropped out and such possible bias could be ignored.
Strengths and limitations
To our knowledge, this is the first study to assess effects of a Coping With Depression/Coping With Strain Course (CWD/CWS) based intervention (Cuijpers et al, 2009; Sælid, 2016) which universally addresses positive mental health and quality of life among public high school students.
Another strength of this study is the adapted randomized control trial design (RCT) with mixed model statistical analyses tested for cluster effects. With the delayed intervention design one group functioned as a control group until the intervention was implemented for this group too. In this way all participants who wanted, could benefit from the intervention, only with a period of delay for some. If the two intervention groups had shown similar positive patterns in the development of the outcome measures, this would have been a strong indication of a positive program effect.
Yet another strength is the inclusion of two comparison samples from the UngData. This made it possible to establish how the outcome measures in the experimental samples related to the relevant general population.
Furthermore, compared to similar studies of CWD/CWS-based interventions, the sample size of this study is relatively large with more than 1,600 in the MindPower sample and more than 18,000 in the national UngData.
The outcome measures used in this study are high quality instruments commonly utilized in studies on adolescents. When measures are administered repeatedly, changes over time can be examined. However, the findings might not have generalizability if the school-sample e.g. background variables, organization and school structures, teacher training, differs in great extend from the current sample.
Limitation are that a full-scale fidelity study could not be obtained. This was to some degree compensated by collecting information from the participants. Unfortunately, there are indications that this information was not 100 percent trustworthy. Therefore, we do not know for sure how exactly the MindPower program manual was followed. On the other hand, we have no indication that this was a widespread problem.
Technical problems with e-mail addresses and with opening of the questionnaires lead to a high level of attrition. Fortunately, the complete responders on the last assessment were not significantly different from those who dropped out. Hence possible sample bias due to selective attrition at this point may be ignored.