Facilitator suggestions to increase cultural relevance of an evidence-based intervention: A qualitative analysis

doi:10.21203/rs.3.rs-5314428/v1

Download PDF

Research Article

Facilitator suggestions to increase cultural relevance of an evidence-based intervention: A qualitative analysis

https://doi.org/10.21203/rs.3.rs-5314428/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background. Once an intervention is deemed “evidence-based,” it is often disseminated across contexts, settings, and populations beyond those originally tested. The cultural relevance or “fit” of an evidence-based intervention (EBI) in diverse local contexts is a debated topic, but few studies have explored facilitators' perceptions. This study examines facilitators' suggestions for improving the cultural relevance of a universal preventive EBI widely disseminated in middle schools for over 30 years.

Method. Using process evaluation data from a dissemination project (2019-2022) in 363 schools across 16 US states, 566 open-ended responses from EBI facilitators were analyzed using the Framework Method (i.e., a content analysis approach).

Results. Five broad themes emerged among 758 unique suggestions: 1) social or technological relevance (52% of codes), 2) teacher or school-related considerations (13% of codes), 3) student engagement (10% of codes), 4) adaptations for diverse students (9% of codes), and 5) curriculum mechanics (4% of codes). Facilitators overwhelmingly suggested improving cultural relevance by updating social and technological aspects to reflect current trends in substance use, youth culture, and implementation.

Discussion. The study underscores a component of cultural relevance that is particularly relevant to on-the-ground implementers. Feasible ways for EBIs to stay current on social trends and technology to reflect evolving trends in risk behaviors, youth culture, and implementation are essential to increase or maintain the cultural relevance of an EBI. Addressing these areas is crucial for enhancing the long-term adoption, fidelity, and effectiveness of EBIs across diverse populations and evolving cultural landscapes.

Evidence-based intervention

scale-up

cultural relevance

schools

Evidence-based interventions (EBIs) refer to specific programs, practices, or strategies that are grounded in empirical evidence and tested in high-quality research studies, such as randomized controlled trials (RCTs). Well-designed and well-implemented RCTs (Gottfredson et al., 2015; Steeger et al., 2021) have confirmed an array of universal, targeted, and indicated preventive interventions for youth to be effective in promoting protective factors and reducing problem behaviors, as well as maintaining positive results over time (Fishbein & Sloboda, 2023). As a result, governmental agencies and foundations have instituted funding mandates requiring the use of evidence to inform the adoption of interventions, which has facilitated the wide-scale dissemination of EBIs (Axford et al., 2022; Williford & Jenson, 2024).

As EBIs are scaled up (i.e., widely disseminated), fidelity of implementation often wanes (Chambers et al., 2013; Dusenbury et al., 2005; Ringwalt et al., 2008). This is because as EBIs move from evaluations (typically in specific contexts where researchers supervise implementation) into dissemination phases (e.g., more diverse contexts where less supervision of implementation occurs), facilitators frequently modify elements of an EBI to fit their local context and needs (Balis et al., 2021; Castro et al., 2004; Stirman et al., 2019). When EBIs are disseminated beyond their original development to varying populations and contexts, assessment of their usefulness and fit can help researchers and practitioners understand factors related to intervention engagement, generalizability, and sustainability. Assessing EBI facilitators’ perspectives of the cultural relevance of an EBI is important because of their direct experience implementing EBIs with youth in diverse contexts, and these perspectives are rarely captured in the literature. Thus, the goal of this study is to explore suggestions from intervention facilitators for how to improve the cultural relevance of a curriculum-based EBI that has been widely disseminated in schools for more than 30 years (Botvin LifeSkills Training).

Cultural Relevance of EBIs in Prevention and Implementation Science

Recently, there have been increasing efforts in prevention and implementation science to understand the role of culture in EBIs, especially universal EBIs initially designed for broad applicability (Balis et al., 2021; Barrera et al., 2017; Bo et al., 2023; Rowe & Trickett, 2018; Shapiro et al., 2024; Steinka-Fry et al., 2017). While deeming interventions as “evidence-based” can strengthen decision-making for communities regarding which interventions to adopt, questions remain about whether an EBI is a good fit for diverse contexts. Most universal prevention programs are generic programs developed broadly for popular youth culture; however, this is heavily influenced by the “majority” group (i.e., White, middle-class) (Rowe & Trickett, 2018; Shapiro et al., 2024). Concerns have arisen about the widespread dissemination of such universal interventions that were largely developed with a majority population but uncritically applied to others (Murry et al., 2024; Pina et al., 2019; Thier et al., 2020). Adding to these questions, many widely-disseminated EBIs have been tested primarily among White, suburban samples of youth and trials inconsistently report socio-demographics of the sample (Buckley et al., 2023; Rowe & Trickett, 2018; Shapiro et al., 2024). Such concerns have motivated questions about the influence that culture (e.g., race, ethnicity, locale, school setting, and English language proficiency) can have on EBI engagement and acceptability and whether universal EBIs are effective across diverse groups and contexts (Castro & Yasui, 2017).

The growing attention and importance of cultural relevance in prevention and implementation science is exemplified foremost by the body of work around culturally specific interventions (CSIs). Substantial guidance exists on developing culturally affirming interventions, and numerous researchers note that cultural adaptation is necessary when implementing universal interventions (Buckley et al., 2023; Jones & Neblett, 2016; Shapiro et al., 2024). CSIs are developed or adapted from an existing EBI for a specific cultural group (Bo et al., 2023). These interventions consider the values, beliefs, and traditions of a particular group to tailor content in a way that is responsive to race, ethnicity, gender, economic status, and geographic location (Bo et al., 2023; Jackson, 2009; Murry et al., 2024; Thier et al., 2020). Culturally-specific adaptations of an EBI can occur along a continuum ranging from minor surface level adjustments of images and language to deeper structural changes that incorporate the targeted audience’s beliefs, values, and lifestyles (Balis et al., 2021; Castro et al., 2004). Despite this spectrum of adaptations, many believe that for a CSI to be effective, it must address deeper level adjustments with historical and societal components integrated into the design, implementation, and review of an intervention (Jackson, 2009).

Many EBIs, however, are not explicitly tailored to a specific group. As mentioned above, universal programs are intended for broad applicability. As universal EBIs are widely disseminated, they are commonly modified by communities and intervention facilitators (e.g., (Combs et al., 2022; Holleran Steiker et al., 2008; Miller-Day et al., 2013; Pankratz et al., 2006; Ringwalt et al., 2004). Such modifications may include facilitators’ impromptu adaptations (or local adaptations) of the intervention during real-world delivery (e.g., accommodations for limited English proficiency, shortening session to fit class time) with aims to increase student participation and engagement with the program (Barrera et al., 2017). Prevention and implementation scientists have cautioned facilitators against modifying EBIs, because such changes may interfere with a program’s core components and jeopardize effectiveness (Dusenbury et al., 2005; Gottfredson & Gottfredson, 2002; Ringwalt et al., 2008). However, more recently, the field has shifted to discussing modifications as essential for increasing cultural relevance, adoption, and sustainment of EBIs (Castro & Kessler, 2021; Castro & Yasui, 2017; Gonzales, 2017). Thus, rather than prohibiting modification, recent research now provides guidance on how to document such adaptations (Stirman et al., 2019) and which types of adaptations will likely not jeopardize core components (Balis et al., 2021).

While the academic literature provides a lively discussion regarding the cultural relevance of widely-disseminated universal EBIs and their acceptable modifications (Castro & Yasui, 2017), few studies explore facilitators’ perceptions of the cultural relevance of EBIs or their suggestions to increase cultural relevance. Qualitative data that are better able to capture the nuances and complexities of this topic are even rarer in published literature. Facilitators can offer particularly helpful insights because they have direct experience implementing EBIs in diverse contexts. Though numerous studies quantitatively document that EBI facilitators frequently modify or adapt universal EBIs to better fit their context and needs (Holleran Steiker et al., 2008; Miller-Day et al., 2013; Pankratz et al., 2006; Ringwalt et al., 2004), few research studies seek insights from these on-the-ground implementation partners. Of the limited research, Ringwalt and colleagues (2004) quantitatively investigated teachers’ reasons for adapting a substance use prevention program in schools with non-White student populations, and they found that EBI facilitators in schools with more racial and ethnic minoritized students were more likely to adapt curriculum to meet their students’ needs and specific context. The authors concluded that an effort should be made to understand why and in what ways teachers are dissatisfied with curricula, and how the curricula may fail to address prevention of substance use and other problem behaviors in a manner that is culturally sensitive or responsive to students’ learning styles (Ringwalt et al., 2004). Schools in particular are increasingly heterogenous contexts with student populations becoming more diverse (e.g., across ethnic, social, economic, and organizational diversity), and schools have varying priorities and availability of resources (Rowe & Trickett, 2018). Thus, insights from facilitators of a universal EBI regarding the cultural relevance in these heterogeneous contexts is valuable to dissemination efforts.

Current Study

Intervention facilitators (e.g., teachers, counselors) are often underutilized sources of data and may provide important perspectives on the cultural relevance of universal EBIs that are widely disseminated, as they are delivering EBIs in diverse contexts and with diverse youth. The current study speaks to this gap in the literature by examining facilitators’ suggestions for improving the cultural relevance of a universal EBI that has been disseminated widely across diverse settings. The Botvin LifeSkills Training (LST) Middle School curriculum is a universal prevention program for middle school-age students typically facilitated by classroom teachers to promote personal self-management skills, social skills, and drug resistance skills (Botvin & Kantor, 2000). The program aims to reduce substance use, and various RCTs demonstrate that LST decreases use of tobacco, alcohol, and marijuana, with effects sustained through high school (Botvin et al., 1995; Botvin et al., 2006; Spoth et al., 2002). LST has been tested among racial and ethnic minoritized youth and across urban/rural settings (Botvin et al., 2006; Spoth et al., 2002). Furthermore, dissemination efforts have shown LST to be scalable. As of 2017, LST had been adopted by over 1,200 communities in the United States serving more than one million youth with a benefit-cost ratio of $15.97 for every $1 spent (Hawkins et al., 2016; Washington State Institute for Public Policy, 2023). Lastly, LST has been disseminated internationally in countries such as Italy (Velasco et al., 2017), China (Huang et al., 2019), Taiwan (Lee et al., 2020), and Iran (Jamali et al., 2016), further demonstrating the wide reach of the program.

Our study uses process evaluation data collected from a national dissemination of the LST Middle School program (https://www.lifeskillstraining.com/) across 16 US states to examine suggestions for cultural relevance from middle school teachers who implemented this school-based EBI. Specifically, our research question was: What do facilitators of a curriculum-based universal prevention program delivered in schools suggest to increase cultural relevance?

LST Dissemination Project

The LST middle school program is a classroom-based intervention, implemented in either grades 6–8 or 7–9, aimed at preventing or reducing substance use, violence, and other problem behaviors. The program consists of 30 core units, divided into three levels that are taught in sequence over three consecutive school years. In the first year, 15 foundational sessions teach core concepts and skills related to self-management, drug resistance, and general social skills. In the second and third years, 10 and five booster sessions (respectively) build on those foundational sessions, allowing time to apply skills to real life, age-appropriate scenarios. In addition to its core sessions, the program includes a total of nine violence prevention lessons and one prescription drug abuse prevention module that teachers have the option to include over the three years of the program. Lessons are intended to be delivered in 45–60-minute class periods at least once per week, during consecutive weeks, and in the prescribed order. While LST lessons can be taught in schools by any individual trained in the curriculum, lessons were usually taught by classroom teachers or school staff in a variety of content areas (e.g., health, social studies, science, math, language arts, school counselors).

Schools were provided with LST curriculum materials, training, process evaluation reports, technical assistance, and sustainability workshops at no cost. An initial one- or two-day training workshop was provided free of cost; training participation was required of all new LST instructors each year of implementation and was encouraged for school administrators and other support staff. Additionally, teachers were offered the option of attending a one-day refresher training in following year(s) and technical assistance was provided throughout the project, including annual visits by the grant’s representatives trained in the LST model to discuss implementation progress and problems. A University Institutional Review Board confirmed that no ethical approval was required for this study due to the exclusive use of retrospective data that were part of routine process evaluation.

Sample

The dissemination project included 122 school districts (representing 363 schools) that participated in a three-year grant to support implementation of LST between academic years 2019–2020 and 2021–2022. Following LST implementation in each academic year of the grant, LST facilitators (i.e., teachers) were invited to complete a survey assessing fidelity of implementation. Over the three academic years, 1,950 teacher feedback surveys were completed. Because teachers could teach LST for all three academic years, an individual teacher could contribute up to three surveys. This analysis includes facilitator surveys that included open-ended responses related to cultural relevance of the program. Of the 1,950 surveys completed over the three years, 566 surveys with open-ended responses were included in our analysis. As shown in Table 1, this sample of LST facilitators with open-ended responses were experienced in teaching, and primarily identified as White (69.1%) and female (68.4%). Differences between those who provided an open-ended response and those who left the item blank were tested: responders had statistically significant lower ratings on a cultural relevance scale of LST (1 = poor to 5 = excellent) than those who skipped the item (M_skipped = 3.38, M_responded = 3.18, p < .001); responders to the open-ended items were more likely to be female and American Indian/Native American; and, responders and non-responders to the open-ended items were similar in regard to identifying as White or Latinx, years of experience, and age.

Table 1

Facilitator Sample Descriptives (N =566)

% / Mean (SD)

Age

20-29

30-39

40-49

50-59

60-69

70-79

Years of experience teaching

Gender

Male

Female

Non-binary

Race/Ethnicity

American Indian

Asian

Black

Hispanic

Multi-racial/ethnic

Other

White

Prefer not to answer

Rating of LST Cultural Relevance (scale 1-5)

How culturally relevant-LST Lessons

How culturally relevant-LST Activities

How culturally relevant-LST Behavioral Rehearsals

12.9%

26.7%

33.2%

18.2%

8.0%

0.9%

14.4 (9.4)

29.2%

68.4%

0.2%

0.4%

0.7%

16.6%

4.2%

3.9%

0.4%

69.1%

4.8%

3.31 (1.06)

3.15 (1.06)

3.07 (1.06)

Of the 363 schools in the dissemination project, 326 schools had a teacher within their district respond to the open-ended items. Thus, we report the school demographics among this sample of schools with district representation in the qualitative analysis sample. Descriptive statistics among all schools in the dissemination project and those represented in this qualitative sample were nearly identical on each sociodemographic variable. The analysis sample of schools (N = 326, Table 2) covered the full 16-state footprint of the dissemination project and was diverse in terms of urbanicity, socioeconomic indicators, race and ethnicity, and region of the US. On average, over half of the students in public schools received free or reduced-price lunch (54.6%); schools were distributed across urban (23.6%), suburban (40.4%), and rural (36.0%) areas; and, about half of students (51.3%) in these schools identified as White (see Table 2).

Table 2

*School Demographic Characteristics (N = 326)*
Characteristic	n schools	%
Locale
Urban	74	23.6%
Suburban	127	40.4%
Rural	113	36.0%
Region
Midwest/West	34	10.4%
Northeast	61	18.7%
Southeast	231	70.9%
School type
Public	314	96.3%
Private	12	3.7%
Student body average across schools:
Students receiving FRL	283	54.6%
Race and ethnicity
American Indian	313	0.09%
Asian	313	2.3%
Black	313	24.7%
Hispanic	313	17.6%
Multi-racial	313	3.9%
White	313	51.3%
Note: FRL = free and reduced-price lunch

Measures

Following three Likert scale items about the cultural relevance of the curriculum (reported in Table 1), teachers were asked the open-ended question, “Please list any suggestions you may have for improving the cultural relevance of LST.” A definition of cultural relevance was not provided in this open-ended question to give teachers autonomy and flexibility in defining the concept based on their lived experiences and diverse school settings. The final survey item was also open-ended and asked for any additional comments or suggestions; responses related to fit or relevance of LST to their students or context were included in the analysis.

Analysis

The Framework Method (Gale et al., 2013), an approach within thematic or content analysis, was used to analyze these data. This method is often used with large sets of qualitative data and with open-ended responses to specific questions, as in this study. The research team first engaged in open coding whereby two coders read each response and independently applied labels (i.e., codes) that described each unique idea within the response to approximately one-quarter of the responses. After independently coding responses, coders met to reach a consensus for each code. Any responses that could not reach consensus were brought to the research team and first author. Then, the research team met to discuss codes and agreed on a working set of codes (i.e., a framework or codebook) and the framework was applied to the remaining responses. The research team met multiple times while coding the remaining responses to discuss the framework and to add to and refine codes. After all responses were coded, a second round of coding (i.e., axial coding) was conducted to identify characteristics of and differences between the data and to identify broad categories of theoretical concepts. In this second round of coding, all responses and codes were reviewed (by a coder different than the original coder) to ensure the code was still accurate and consistent given the iterative process to develop the framework, and to ensure application of the broad themes. Lastly, the research team discussed the final thematic framework and interpretation of the data.

Each open-ended response had between one and three unique suggestions or units of analysis. Thus, the 566 surveys with open-ended responses yielded 758 unique suggestions or ideas (i.e., codes) that were relevant to the research question of suggestions for increasing the cultural relevance of the LST intervention. In addition to favorable comments and responses that did not align with broad themes, five salient themes were identified: 1) social or technological relevance, 2) teacher- or school-related considerations, 3) student engagement, 4) adaptations for diverse students, and 5) mechanics of the curriculum materials (Table 3). The largest proportion of codes (52%) related to updating social and technological components of the curriculum (n_codes=393). This included general or vague comments about updating materials, as well as specific suggestions for integrating more technology, updating material to fit current trends in drug use, updating visuals and scenarios to be more current, and updating the content to include new topics that youth navigate today (e.g., social media, technology).

Table 3

*Cultural Relevance Themes and Example Quotes*
Theme	Description	N	% of codes	Example quote 1	Example quote 2
Social and technological relevance, n_codes = 393, 52% of codes
Update materials (general)	General and usually vague comments about materials being outdated and needing updates.	62	8.2%	The series needs to be updated with modern language and situations.	Updated data. More dynamic workbooks. More variety in activities.
Better use of technology	Update curriculum to include PowerPoint slides, online resources, visual aids, and online accessibility.	109	14.4%	The LST program desperately needs to incorporate technology. Create videos, have interactive tests and quizzes, create digital course materials, have up to date data on the issues affecting young people. Make it exciting and fun so that teachers can use the materials you all provide versus just using the LST books for temporary supplements.	There really needs to be something other than a pencil/paper option. The vast majority of my students don't bring pencils, and they don't return the ones I lend them.
Outdated drug-related materials	Update content and materials to fit current trends in drug use (e.g., vaping, marijuana).	74	9.8%	Our students vape, rather than smoke. It is understood that addiction to nicotine, no matter how it is delivered, is the core issue. However, the constant references to "smoking", rather than "vaping" is becoming ever more notable for missing the mark with our students.	With marijuana laws changing, I believe it is critical that updated information/perceptions on marijuana be introduced. I also think it is extremely important to cover vaping, Juuling, etc. in more depth.
Outdated videos, visuals, scenarios, etc.	Update visuals in curriculum and activities (e.g., videos, statistics, behavioral rehearsals).	98	12.9%	Some materials and scenarios seem a bit dated. The smoking and biofeedback video has great content, but the video is outdated to the point that it is distracting to students.	Update some of the examples used in the workbook. For example, in one lesson the term "mp3 player" was used. Most of the students had no idea what an mp3 player was.
Outdated content	Update to include other content that's pertinent to today's youth (e.g., social media, technology, texting, cyberbullying, video games, etc.).	50	6.6%	More examples involving technology and social media would help the content feel more relevant to them.	Adding social media safety and etiquette is a needed addition.
Teacher/school considerations, n_codes = 100, 13% of codes
Placement in a class with aligned content	Teachers protective of class time when LST content is inappropriate placement for the class.	33	4.4%	I think the program is beneficial to our students, but I wish the lessons could have been taught in students’ connection classes instead of by content teachers because it reduced time that we could have been working to help build their academic deficits.	When the students hear that LST (or any other class) will be taking away their PE, they immediately shut down and most refuse to even try to participate. This is not an issue with the program itself, more so the school’s decision to teach it during PE times.
Logistics of when and where	Logistics such as the class size, time of year, or location of class that are largely outside of teacher control.	14	1.8%	So our district required us to teach LST at the end of the first semester. It is almost too late. We should teach it as soon as they start 6th grade.	I believe this would work better with a classroom teacher(science) who only has up to 25 students or so.
Age-related placement concerns	Content being inappropriate for age of students (too young or too old).	32	4.2%	There are some scenarios in the LST lessons that aren't relevant to the students' experience at the 6th grade level, but may be as the students enter junior high. For example, few have been offered cigarettes, but vaporizers, marijuana, etc. may be prevalent at the next grade level.	The language in the book needs to be reassessed. Sometimes the book was talking down to them, as if they weren't smart. My students felt like they were being treated and spoken to like babies if it was coming from the book.
Teacher autonomy	Inflexibility of curriculum preventing teacher's adaptation to their class, or that cultural relevance is dependent on. the teacher and how "bought-in" they are.	13	1.7%	The LST program needs to be more flexible when it comes to timing and implementation. Each school, grade level, and team of teachers is going to have their own creative ways to teach the curriculum within their own parameters. Teachers need to have the autonomy to teach the program in a way that works best for them individually.	I like the program and I think it will prove very useful to the students as they mature. I wish the implementation of the program allowed for a little more flexibility because it is difficult for me to adapt it to the needs and tastes of my students. Sometimes, I felt like I needed to go off-script or could have even used a little more script to effectively make the points.
Too many demands on teacher	Competing demands on teachers prevents them from delivering program with cultural relevance.	8	1.1%	There are some good elements to the program, but it is really tough to implement a program that requires this much time in our very packed schedules with many other demands on our teaching time. Thanks for considering my concerns.	I really liked the program, but the expectations and material are basically another prep. I know we had many resources, but it still takes time to sift through everything and flip from teacher book to student book and remember it all!
Student engagement, n_codes = 75, 10% of codes
Activities/ Behavioral Rehearsals	Comments related to improving activities, having more interactive activities or strategies to avoid awkwardness in behavioral rehearsals.	49	6.5%	There weren’t enough fun games, videos, or activities in the module. They should also be built right in rather than you having to look in the back index for an activity. There weren’t enough engaging opportunities for students, especially when many aren’t comfortable with sharing.	Although there were a few activities for student practice, including more hands-on activities will keep student interests. At times, students struggled with writing so hands on would be more beneficial.
Student related	Bored, unengaged	26	3.4%	The lessons content is very relevant to my students, but the lessons were not engaging to my students	Many students stated that they did the same lessons in the Level 1 LifeSkills training in 6th grade. They absolutely dreaded the lessons. New material needs to be created for each level so that it seems relevant and new.
Adaptations for diverse groups, n_codes = 69, 9% of codes
Subgroups	Curriculum is not relevant to specific groups or the local community including youth from various socio-economic levels and youth who have experienced trauma.	50	6.6%	This program appears, in my opinion, to be geared toward suburban, middle class, white students. Many of my students have witnessed shootings, ODs, and other violent crimes. The things in the program have merit but are not deep enough for these kids. They need de-escalation strategies, skills for removing yourself from unsafe situations, how to deal with gun violence, coping with the loss of loved ones, foster care trauma.	I would only suggest that some of the scenarios be more realistic for children from challenging demographic backgrounds. Some of the children face more bad experiences in a morning than most of us experience in a year. Having the scenarios take into consideration the serious drug issues we have in our area may leave the children even more prepared when they leave us.
Language	Translations to other languages (mostly Spanish)	19	2.5%	There needs to be a Spanish version for our ESL kids who are not proficient in the English language.	Make LifeSkills available in other languages besides English.
Mechanics of curriculum materials, n_codes = 32, 4% of codes
Quantity of material	Too many/few lessons, too long of lessons, or not having sufficient time.	22	2.9%	I think that it could be expanded into a full curriculum for Health classes. These lessons and activities take one class period. I think that if you expanded them and they took a duration of the semester, I think it would be more appealing to everyone.	More relative information as far as facts, some of the lessons where outdated. Lesson lengths vary from long to extremely short, finding a happy medium would be nice.
User-friendliness	Cohesiveness between teacher manual and student workbooks	10	1.3%	I would suggest that a small team go through the teacher manual with the student materials and check for seamless engagement between the two. Sometimes things are skipped or left out in the manual that makes for lost time and lower engagement from students.	We really struggled with the student workbook and the teacher guide not following each other very well. It was confusing to figure out at times.
Favorable comments, n_codes = 55, 7% of codes
Favorable comments	Teachers believe it is culturally relevant.	55	7.3%	I think for my state and the area in which I teach I feel that all the lessons are very relevant, especially the age group!	I thought the curriculum was on point in this regard and it was rather eye opening how much relevant experience these students are walking around with and are willing to share with the group.
Other, n_codes = 34, 5% of codes
Other	COVID-19, reading level, training-related	34	4.5%	With Covid restrictions some classes are 5 students or less, perhaps adding smaller group activities. The additional resources for quarantined or at home students would helpful as well.	The activities and behavioral rehearsals could use more training to understand how to effectively present them.

Teacher or school-related considerations (n_codes= 100; 13%) included suggestions about the placement of LST with other academic content, the logistics of when and where to implement LST in a busy school day, and the appropriateness of the content for the age or grade of students. A subtheme arose within this theme noting that teacher autonomy or initiative to make a curriculum culturally relevant was critical and that too many demands may prevent teachers from implementing well. The student engagement theme (n_codes=75; 10%) arose from codes about the need for improving or having more interactive activities and student boredom. Themes related to diverse groups (n_codes=69; 9%) included that LST often lacks relevance for youth who experience trauma or do not speak English as their first language. Mechanics of the curriculum (n = 32; 4%) referred to the quantity of material (e.g., too short or too long) and user-friendliness of the materials.

Teachers also used the open-ended question to express that they felt the EBI was culturally relevant or easy to implement within their context (n_codes=55, 7%). Several codes fell into an “Other” category (n_codes=34, 5%) due to the small number of the codes that did not align with the other emerging themes. These “Other” codes included comments about the COVID-19 (Coronavirus Disease 2019) pandemic (n_codes=16), training needs for delivering the program (n_codes= 14), and reading levels of students (n_codes=4).

Extensive discussion and research exist on the topic of cultural relevance of EBIs, especially as they are scaled up and disseminated beyond contexts and populations for which they were originally developed and tested. However, this body of research has rarely included facilitators’ ideas for enhancing the cultural relevance of widely-disseminated EBIs. This study examined open-ended responses from LST implementation facilitators (i.e., teachers, counselors) across 326 schools in 16 US states, including schools in diverse urban, suburban, and rural settings and among diverse student sample characteristics. Five salient themes emerged from teachers across these school contexts: social and technological relevance, teacher or school related considerations, adaptations for diverse students, engagement of students, and mechanics of the curriculum materials.

The overwhelming theme from this sample of facilitators was that the current social and teen culture matters, with over half of suggestions recommending updates to the EBI’s social relevance or technology. Thus, results from facilitator suggestions highlight that maintaining “cultural relevance” may include ongoing attention to social trends and regularly updating intervention materials in a real and useful way over time. Given how prevalent this theme was in a sample of facilitators in diverse school settings, it seems that reflecting current trends in substance use, social problems, and technology is essential regardless of student demographics or school context.

After updating materials and content to fit current trends, the next largest theme was teacher and school considerations, including LST placement concerns (i.e., alignment with content of class, logistics, and age appropriateness). While this may not seem overtly related to “cultural relevance,” it does speak to school context and culture and whether LST is a good fit for the local context. This theme also highlights the importance of how EBIs are introduced into communities and understanding the constraints and demands already upon the community, school, or teacher. As noted in the emerging themes, teachers often experience too many demands under time constraints. Thus, support is needed to help facilitators modify intervention materials to the context of their students and school. Suggesting that teachers adapt activities, scenarios, and role plays to their local context with minor surface level or moderate adaptations (e.g., Balis et al., 2021) may not be feasible in many situations. EBI developers have an important role to support modification options that address cultural relevancy and context but maintain fidelity to the core components of the intervention. EBI developers and intervention implementers should work together to determine which culturally-sensitive adaptations are needed, the feasibility of adaptions, and documentation of processes (Castro & Yasui, 2017; Kumpfer et al., 2017). This will be an ongoing process as adaptation guidelines have changed over time, and who is responsible for guiding and implementing adaptations will vary by context and specific EBI.

The prevention and implementation science literature has largely conceptualized cultural relevance as adaptations for populations who are different than the group for which the intervention was originally developed or tested (typically “different” by sociodemographic characteristics such as race, ethnicity, or gender identity) (Barrera et al., 2017; Bo et al., 2023; Shapiro et al., 2024). This student-level sociodemographic focus on cultural relevance and adaptation is critical, and it is essential to reach equitable intervention effects for population-level impacts (Gonzales, 2017). This sample of facilitators in the current study also highlight that “cultural relevance” may include ongoing attention to social trends, regularly updating intervention materials, and context specific considerations (e.g., classroom placement, instructional time, length of sessions). Some research has noted the importance of this expanded view of cultural relevance, particularly in relation to maintaining participant engagement and sustainability when scaling up in natural service delivery systems (Gonzales, 2017). The current study emphasizes that for facilitators, who are direct implementers, these pragmatic considerations are particularly critical elements.

While the current study examined facilitators of a single EBI, the need to address social and technological relevance and specific contextual constraints applies to many EBIs, particularly curriculum-based interventions delivered in schools. Developing and testing an EBI is an enormous investment of money and time. If the prevention and implementation science fields aim to be successful in establishing ongoing EBI implementation in communities and schools well beyond their development, a feasible way of updating intervention materials is critical. The best available EBIs should include updated intervention content, examples, and activities and use materials that are compatible with current teaching/facilitating methods (e.g., tablets instead of notebooks, PowerPoint slides). With these updates, schools will be more likely to adopt EBIs, facilitators will have more success implementing effectively, and students will be more engaged. These are important concepts, as adoption, fidelity of implementation, and participant engagement are prerequisites for EBIs to effectively change behaviors in the targeted population (Barrera et al., 2017; Holleran Steiker et al., 2008; Kumpfer et al., 2017). Understanding the core components of curriculum-based EBIs will be helpful for facilitators and developers to adapt to social and cultural changes over time as well as to adapt to different populations. Further, creating greater online accessibility may facilitate such updates, and including resources that provide clear examples of adaptation for specific groups of youth and for current social realities could improve cultural relevance and ensure fidelity-consistent adaptations and modifications.

Recent research provides guidance to EBI implementers, developers, and researchers on how to document adaptations and the types of adaptations that likely will not jeopardize core components (Balis et al., 2021; Steinka-Fry, et al., 2017). For example, Balis et al. (2021) provide a stop light metaphor to guide adaptations. In this model, adaptations are categorized by the level at which they modify the surface level or core components of the program, with adaptations classified as either green light, yellow light, or red light. Within this framework, green light adaptations are minor and can include tailoring the cultural appropriateness of language, pictures, scenarios, or relevant health statistics, whereas yellow light adaptations often involve adding or substituting activities or modifying the delivery of the program. While green and yellow light intervention adaptations frequently still allow for effective youth outcomes, red light adaptations are categorized as such to caution implementors against changing core components of the program, deleting or removing content, or changing the key focus of sessions or whole programs (Balis et al., 2021).

Though many suggestions from facilitators in the current study did not fit into the stop light metaphor because they were not related to the curriculum (e.g., teacher or school considerations), those that did were generally green and yellow light changes. For example, shifting from a printed handbook to an electronic version that could be done on a tablet, updating visuals, and including more interactive activities would generally be considered green light changes. Further, incorporating different substances/drugs into a module or creating scenarios that are inclusive of broader identities and experiences would generally be considered yellow light changes. We observed fewer facilitator comments related to red light changes (e.g., shortening or deleting lessons, removing program components), though it was present in the theme “quantity of material” that described issues of too many lessons, too long of lessons, or not having sufficient time given the time constraints teachers often face. Formally adapting EBIs requires numerous resources. Particularly for universal interventions, allowing facilitators the flexibility to make green and yellow light changes, without having to test an entirely new adapted program, may provide a feasible option for enhancing cultural relevance.

It should be noted that in response to requests for online formats and activities, the developers of LST created basic slide shows for the lessons, as well as an online version of the intervention (i.e., eLST; (Griffin et al., 2022)), and an interactive game called Galaxia. Importantly, this version may meet some facilitator and student needs or requests for an interactive, online program format, however, they have not been rigorously evaluated. Also, some components in the original EBI in-person format (e.g., peer-to-peer learning) are not possible with many online formats, and the intervention costs (including a new license each year for each student) may result in additional barriers for sustainability. Thus, changes in formats to web-based delivery and use of other technology may have trade-offs with the originally tested and effective intervention format. Nonetheless, e-LST and other online versions of programs stemming from EBIs need to be sufficiently evaluated for program efficacy, particularly since the COVID-19 pandemic when many EBIs transitioned to online or hybrid modes of delivery and/or facilitator training due to necessity (Buckley et al., 2022).

Strengths and Limitations

Strengths of this qualitative study include examining under-researched facilitator suggestions for increasing cultural relevance of a well-known, universal EBI and using a large, diverse, sample of facilitators in schools across 16 US states. At the same time, key limitations should be noted. First, we did not provide LST facilitators with a specific definition of cultural relevance and allowed this to be open for each facilitator to interpret for their own context. As facilitators of an EBI, teachers have unique insights into the challenges to implementing programs for their students. For some teachers and schools, cultural relevance may reflect racial identity, and for others it may be related to geographical location, effects of historical events in their community, or many other considerations. Many teachers are trained to teach in a culturally-sensitive manner. In this way, teachers have expertise on topics of cultural relevance and competence within their own context. Second, teachers were less racially and ethnically diverse than the student bodies they reflected (e.g., 69% of teachers identified as White compared to 52% of students who identified as White), which may limit their ability to understand or relate to what an EBI’s cultural relevance means to their students. Still, EBI facilitators in this study provide valuable insights into their school and students that developers and researchers cannot provide. Third, while this is a large qualitative sample, it is not nationally representative, and thus results cannot be generalized broadly to all EBIs or all EBI facilitators.

Among a large group of facilitators implementing a widely-disseminated EBI, participants overwhelmingly suggested improving program cultural relevance by updating social and technological aspects to reflect current trends in substance use, youth culture, and implementation. Only a small percentage of suggestions focused on sociodemographic factors, which is the primary focus of cultural relevance and adaptation in academic literature. Thus, this sample highlighted a component of cultural relevance related to context that may be overlooked by researchers and developers, but particularly pertinent to on-the-ground implementers. The study underscores the importance of EBI implementers, developers, and researchers working together to update materials and provide clear, fidelity-consistent adaptation guidelines. Addressing these areas is crucial for enhancing the long-term adoption, fidelity, and effectiveness of EBIs across diverse populations and evolving cultural landscapes.

Author Contribution

Concepts and design (KMC, VG, CS, ALC, MAL); data coding, management, and analysis (KMC, VG, ALC, BLH, LS, SZ); all authors were involved in drafting of manuscript as well as intellectual contributions, reviewing, and critical editing of manuscript content. All authors have read and approved the final manuscript.

Acknowledgement

The authors would like to thank Dr. Pamela R. Buckley for consultation and critical read of the manuscript. Process evaluation data utilized in this study were collected through the support of a prevention dissemination grant from the Altria Group; however, this funder had no influence on the hypotheses, analyses, or interpretation and reporting of these results.

Axford, N., Morpeth, L., Bjornstad, G., Hobbs, T., & Berry, V. (2022). “What works” registries of interventions to improve child and youth psychosocial outcomes: A critical appraisal. Children and Youth Services Review, 137, 106469. https://doi.org/https://doi.org/10.1016/j.childyouth.2022.106469
Balis, L. E., Kennedy, L. E., Houghtaling, B., & Harden, S. M. (2021). Red, yellow, and green light changes: Adaptations to extension health promotion programs. Prevention Science, 22, 903-912. https://doi.org/10.1007/s11121-021-01222-x
Barrera, M., Berkel, C., & Castro, F. G. (2017). Directions for the Advancement of Culturally Adapted Preventive Interventions: Local Adaptations, Engagement, and Sustainability. Prevention Science, 18(6), 640-648. https://doi.org/10.1007/s11121-016-0705-9
Bo, A., Goings, T. C., Evans, C. B. R., Sharma, A., Jennings, Z., Durand, B., Bardeen, A., & Murray-Lichtman, A. (2023). Culturally sensitive prevention programs for substance use among adolescents of color: A systematic review and meta-analysis of randomized controlled trials. Clinical Psychology Review, 99, 102233. https://doi.org/https://doi.org/10.1016/j.cpr.2022.102233
Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. JAMA, 273(14), 1106-1112. https://doi.org/doi:10.1001/jama.1995.03520380042033
Botvin, G. J., Griffin, K. W., & Nichols, T. D. (2006). Preventing youth violence and delinquency through a universal school-based prevention approach. Prevention Science, 7(4), 403-408. https://doi.org/DOI 10.1007/s11121-006-0057-y
Botvin, G. J., & Kantor, L. W. (2000). Preventing alcohol and tobacco use through life skills training. Alcohol Research & Health, 24(4), 250. https://link.gale.com/apps/doc/A78395687/AONE?u=coloboulder&sid=AONE&xid=762bc35a
Buckley, P. R., Edwards, D., Ladika, A., Steeger, C. M., & Hill, K. G. (2022). Implementing evidence-based preventive interventions during a pandemic. Global Implementation Research and Applications, 2(4), 266-277. https://doi.org/10.1007/s43477-022-00047-2
Buckley, P. R., Murry, V. M., Gust, C. J., Ladika, A., & Pampel, F. C. (2023). Racial and ethnic representation in preventive intervention research: A Methodological Study. Prevention Science, 24(7), 1261-1274. https://doi.org/10.1007/s11121-023-01564-8
Castro, F. G., Barrera, M., & Martinez, C. R. (2004). The cultural adaptation of prevention interventions: Resolving tensions between fidelity and fit. Prevention Science, 5(1), 41-45. https://doi.org/10.1023/B:PREV.0000013980.12412.cd
Castro, F. G., & Kessler, R. (2021). Cultural Factors in Prevention. In W. O’Donohue & M. Zimmermann (Eds.), Handbook of Evidence-Based Prevention of Behavioral Disorders in Integrated Care: A Stepped Care Approach (pp. 51-81). Springer International Publishing. https://doi.org/10.1007/978-3-030-83469-2_4
Castro, F. G., & Yasui, M. (2017). Advances in EBI development for diverse populations: Towards a science of intervention adaptation. Prevention Science, 18(6), 623-629. https://doi.org/10.1007/s11121-017-0809-x
Chambers, D. A., Glasgow, R. E., & Stange, K. C. (2013, 2013/10/02). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8(1), 117. https://doi.org/10.1186/1748-5908-8-117
Combs, K. M., Buckley, P. R., Lain, M. A., Drewelow, K. M., Urano, G., & Kerns, S. E. U. (2022). Influence of Classroom-level factors on implementation fidelity during scale-up of evidence-based interventions. Prevention Science, 23, 969-981. https://doi.org/10.1007/s11121-022-01375-3
Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005, Jun). Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health education research, 20(3), 308-313. https://doi.org/10.1093/her/cyg134
Fishbein, D. H., & Sloboda, Z. (2023). A national strategy for preventing substance and opioid use disorders through evidence-based prevention programming that fosters healthy outcomes in our youth. Clinical Child and Family Psychology Review, 26(1), 1-16. https://doi.org/10.1007/s10567-022-00420-5
Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013). Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology, 13(1), 117. https://doi.org/10.1186/1471-2288-13-117
Gonzales, N. A. (2017, 2017/08/01). Expanding the cultural adaptation framework for population-level impact. Prevention Science, 18(6), 689-693. https://doi.org/10.1007/s11121-017-0808-y
Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015, 2015/10/01). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16(7), 893-926. https://doi.org/10.1007/s11121-015-0555-x
Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39(1), 3-35. https://doi.org/https://doi.org/10.1177/002242780203900101
Griffin, K. W., Williams, C., Botvin, C. M., Sousa, S., & Botvin, G. J. (2022). Effectiveness of a hybrid digital substance abuse prevention approach combining e-Learning and in-person class sessions. Frontiers in Digital Health, 4. https://doi.org/10.3389/fdgth.2022.931276
Hawkins, J. D., Jenson, J. M., Catalano, R., Fraser, M. W., Botvin, G. J., Shapiro, V., Brown, C. H., Beardslee, W., Brent, D., & Leslie, L. K. (2016). Unleashing the power of prevention. American Journal of Medical Research, 3(1), 39.
Holleran Steiker, L. K., Castro, F. G., Kumpfer, K., Marsiglia, F. F., Coard, S., & Hopson, L. M. (2008). A Dialogue Regarding Cultural Adaptation of Interventions. Journal of Social Work Practice in the Addictions, 8(1), 154-162. https://doi.org/10.1080/15332560802112094
Huang, C.-C., Chen, Y., Cheung, S., Greene, L., & Lu, S. (2019). Resilience, emotional problems, and behavioural problems of adolescents in China: Roles of mindfulness and life skills. Health & Social Care in the Community, 27(5), 1158-1166. https://doi.org/https://doi.org/10.1111/hsc.12753
Jackson, K. F. (2009). Building cultural competence: A systematic evaluation of the effectiveness of culturally sensitive interventions with ethnic minority youth. Children and Youth Services Review, 31(11), 1192-1198. https://doi.org/https://doi.org/10.1016/j.childyouth.2009.08.001
Jamali, S., Sabokdast, S., Sharif Nia, H., Goudarzian, A. H., Beik, S., & Allen, K. A. (2016). The effect of life skills training on mental health of Iranian middle school students: A Preliminary study. Iran J Psychiatry, 11(4), 269-272.
Jones, S. C. T., & Neblett, E. W. (2016). Racial–Ethnic Protective Factors and Mechanisms in Psychosocial Prevention and Intervention Programs for Black Youth. Clinical Child and Family Psychology Review, 19(2), 134-161. https://doi.org/10.1007/s10567-016-0201-6
Kumpfer, K., Magalhães, C., & Xie, J. (2017). Cultural adaptation and implementation of family evidence-based interventions with diverse populations. Prevention Science, 18(6), 649-659. https://doi.org/10.1007/s11121-016-0719-3
Lee, M.-J., Wu, W.-C., Chang, H.-C., Chen, H.-J., Lin, W.-S., Feng, J. Y., & Lee, T. S.-H. (2020). Effectiveness of a school-based life skills program on emotional regulation and depression among elementary school students: A randomized study. Children and Youth Services Review, 118, 105464. https://doi.org/https://doi.org/10.1016/j.childyouth.2020.105464
Miller‐Day, M., Pettigrew, J., Hecht, M. L., Shin, Y., Graham, J., & Krieger, J. (2013). How prevention curricula are taught under real‐world conditions: Types of and reasons for teacher curriculum adaptations. Health Education, 113(4), 324-344.
Murry, V. M., Bradley, C., Cruden, G., Brown, C. H., Howe, G. W., Sepùlveda, M.-J., Beardslee, W., Hannah, N., & Warne, D. (2024, 2024/01/01). Re-envisioning, Retooling, and Rebuilding Prevention Science Methods to Address Structural and Systemic Racism and Promote Health Equity. Prevention Science, 25(1), 6-19. https://doi.org/10.1007/s11121-022-01439-4
Pankratz, M. M., Jackson-Newsom, J., Giles, S. M., Ringwalt, C. L., Bliss, K., & Bell, M. L. (2006). Implementation fidelity in a teacher-led alcohol use prevention curriculum. Journal of Drug Education, 36(4), 317-333. https://doi.org/10.2190/h210-2n47-5x5t-21u4
Pina, A. A., Polo, A. J., & Huey, S. J. (2019). Evidence-Based psychosocial interventions for ethnic minority youth: The 10-year update. Journal of Clinical Child & Adolescent Psychology, 48(2), 179-202. https://doi.org/10.1080/15374416.2019.1567350
Ringwalt, C., Hanley, S., Vincus, A. A., Ennett, S. T., Rohrbach, L. A., & Bowling, J. M. (2008). The prevalence of effective substance use prevention curricula in the nation’s high schools. The Journal of Primary Prevention, 29(6), 479-488. https://doi.org/https://doi.org/10.1023/A:1020872424136
Ringwalt, C. L., Vincus, A., Ennett, S., Johnson, R., & Rohrbach, L. A. (2004). Reasons for teachers' adaptation of substance use prevention curricula in schools with non-white student populations. Prevention Science, 5(1), 61-67. https://doi.org/10.1023/B:PREV.0000013983.87069.a0
Rowe, H. L., & Trickett, E. J. (2018). Student diversity representation and reporting in universal school-based social and emotional learning programs: Implications for generalizability. Educational Psychology Review, 30(2), 559-583. https://doi.org/10.1007/s10648-017-9425-3
Shapiro, V. B., Eldeeb, N., McCoy, H., Trujillo, M., & Jones, T. M. (2024). Where’s the bipoc blueprint for healthy youth development? The role of scientific omissions in our struggle for science translation and racial equity in the united states. Journal of Prevention, 45(2), 303-321. https://doi.org/10.1007/s10935-024-00771-5
Spoth, R. L., Redmond, C., Trudeau, L., & Shin, C. (2002). Longitudinal substance initiation outcomes for a universal preventive intervention combining family and school programs. Psychology of Addictive Behaviors, 16(2), 129. https://doi.org/https://doi.org/10.1037/0893-164X.16.2.129
Steeger, C. M., Buckley, P. R., Pampel, F. C., Gust, C. J., & Hill, K. G. (2021). Common methodological problems in randomized controlled trials of preventive interventions. Prevention Science, 22(8), 1159-1172. https://doi.org/10.1007/s11121-021-01263-2
Steinka-Fry, K. T., Tanner-Smith, E. E., Dakof, G. A., & Henderson, C. (2017). Culturally sensitive substance use treatment for racial/ethnic minority youth: A meta-analytic review. Journal of substance abuse treatment, 75, 22-37.
Stirman, S. W., Baumann, A. A., & Miller, C. J. (2019). The FRAME: An expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science, 14(1), 1-10. https://doi.org/https://doi.org/10.1186/s13012-019-0898-y
Thier, M., Martinez, C. R., Alresheed, F., Storie, S., Sasaki, A., Meline, M., Rochelle, J., Witherspoon, L., & Yim-Dockery, H. (2020). Cultural adaptation of promising, evidence-based, and best practices: A scoping literature review. Prevention Science, 21(1), 53-64. https://doi.org/10.1007/s11121-019-01042-0
Velasco, V., Griffin, K. W., Botvin, G. J., Celata, C., Velasco, V., Antichi, M., … & Gruppo, L. S. T. L. (2017). Preventing Adolescent Substance Use Through an Evidence-Based Program: Effects of the Italian Adaptation of Life Skills Training. Prevention Science, 18(4), 394-405. https://doi.org/10.1007/s11121-017-0776-2
Williford, A., & Jenson, J. M. (2024). Promoting the Behavioral Health of Children and Adolescents: Evidence-Based Prevention Strategies in Schools, Families, and Communities (2nd ed.). Cognella, Inc. https://titles.cognella.com/promoting-the-behavioral-health-of-children-and-adolescents-9781793585509
Washington State Institute for Public Policy (2023). Updated inventory of evidence-based, research based, and promising practices: LifeSkills Training Benefit-Cost. https://www.wsipp.wa.gov/BenefitCost/Program/37

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Facilitator suggestions to increase cultural relevance of an evidence-based intervention: A qualitative analysis

Status:

Version 1

Abstract

Introduction

Cultural Relevance of EBIs in Prevention and Implementation Science

Current Study

Methods

LST Dissemination Project

Sample

Measures

Analysis

Results

Discussion

Strengths and Limitations

Conclusions

Declarations

Author Contribution

Acknowledgement

References

Additional Declarations

Status:

Version 1