Systematic Review and Meta-Analyses of Mental Health App RCTs: Persuasive Design, Engagement, and Efficacy

doi:10.21203/rs.3.rs-5302304/v1

Download PDF

Article

Systematic Review and Meta-Analyses of Mental Health App RCTs: Persuasive Design, Engagement, and Efficacy

https://doi.org/10.21203/rs.3.rs-5302304/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

This systematic review and meta-analysis examined the efficacy of digital mental health apps and the impact of persuasive design principles on engagement and clinical outcomes. Of 119 eligible randomised controlled trials, 92 studies (n=16,728) were included in the analysis. Results demonstrated that mental health apps significantly improved clinical outcomes compared to control groups (g = 0.43). Apps used between 1 and 12 persuasive design principles (mode = 5). Notably, only 76% of studies reported engagement data. Twenty-five engagement metrics were identified across studies and grouped into 10 categories. Meta-regression and correlation analyses found no significant association between persuasive design principles and app efficacy or engagement. Future research should prioritise standardising and documenting engagement metrics and persuasive design principles; differentiating between engagement with mental health apps and real-world behavioural change and exploring the integration of persuasive design with behaviour change models to more accurately assess their influence on engagement and outcomes.

Health sciences/Health care

Health sciences/Medical research

Mental health care systems face a multitude of serious challenges, including an ever-increasing demand for mental health support, insufficient funding, employment insecurity, high staff turnover, understaffing, and clinician burnout (Lattie et al., 2019; Lipschitz et al., 2022; Lipschitz et al., 2023; Welles et al., 2022). Concurrently, people seeking mental health support frequently encounter barriers to care such as long waitlists, reduced appointment availability, prohibitive financial costs, and limited access to ongoing sessions (Corscadden et al., 2019; Mulraney et al., 2021; Subotic-Kerry et al., 2023). The widespread use of smartphones, along with the growing global demand for mental health care, has driven the development of innovative digital technologies to address unmet mental health needs (Lipschitz et al., 2023; Torous et al., 2020). Digital interventions represent a transformative shift in treatment delivery paradigms (Schueller et al., 2013). They have the potential to alleviate pressures on in-person clinical services and support an increasing population in need of mental health care by providing accessible, cost-effective, and evidence-based mental health treatment (Bond et al., 2023; Engel et al., 2023; Linardon et al., 2024).

Digital mental health apps can be self-guided, fully automated and delivered independent of a clinician (Saad et al., 2021), with the potential to enable convenient access to support while reducing costs (Engel et al., 2023; Gaga et al., 2022; Gentili et al., 2022; ) and improving management of wait-times for clinical services (Cross et al., 2019; Eichsted et al., 2024). These interventions can also complement traditional face-to-face care through ‘blended care’ models, where clinicians integrate technology into their practice to enhance both online and offline components of care (Cross et al., 2023; Erbe et al., 2017; O’Sullivan et al., 2024; Valentine et al., 2020). Moreover, new digital interventions deliver both self-guided and blended care, effectively filling gaps across treatment phases such as waitlist management, blended care, and relapse prevention (Alvarez-Jimenez et al., 2024). The scalability of digital interventions holds promise for delivering personalised treatments to a much larger population than traditional face-to-face services can accommodate (Lipshitz et al., 2023; Valentine et al., 2023). Consequently, evidence-based digital technology emerges as a valuable resource for addressing the disparity between the demand for mental health care and the available supply (Lattie et al., 2019).

Digital mental health interventions are generally well-regarded by clinicians and young people (Bell et al., 2023). They have demonstrated efficacy in both high-prevalence disorders such as depression and anxiety (Gan et al., 2021; Linardon et al., 2024; McCall et al., 2021), as well as complex and severe disorders such as psychosis (Alvarez-Jimenez et al., 2021). Additionally, some interventions have been shown to be cost-effective compared to treatment as usual (Engel et al., 2023). However, low user engagement remains a significant and long-standing issue (de Beurs et al., 2017; Fleming et al., 2018; Kelders et al., 2013; Torous et al., 2020; Lipschitz et al., 2022). In a review of real-world user engagement with popular mental health apps, Baumel et al. (2019) reported an engagement rate of 3.9% by day 15, declining to 3.3% by day 30. In a systematic review conducted by Fleming et al. (2018) examining user engagement in samples of people with depression and anxiety, sustained usage varied widely, ranging from 0.5–28.6%. These low and inconsistent rates of engagement, despite demonstrated positive outcomes in controlled environments (Linardon et al., 2024), raise concerns about the clinical utility and effectiveness of digital mental health interventions in real-world scenarios (Lipschitz et al., 2022).

Overcoming this engagement challenge is an essential component in bridging the gap between potential effectiveness and practical impact (Lipschitz et al., 2023). To be efficacious, digital interventions must enable offline therapeutic actions — desired health behaviours initiated within a digital mental health app that are subsequently adopted into real-world settings (Baumel, 2022). This aligns with Cole-Lewis' (2019) concept of "Big E" (health behaviour engagement) proposed in the framework for digital behaviour change interventions, which emphasises the importance of engagement leading to actual health behaviour changes. According to this framework, the achievement of Big E—the real-world adoption of health behaviours—is contingent on "Little e" (user engagement with the app’s features), which includes interactions with both app elements (e.g., games) and behaviour change techniques designed to influence health outcomes. The framework highlights that real-world behaviour change (Big E) is unlikely without meaningful interaction with the intervention, reinforcing the notion that offline therapeutic actions are a critical component of successful digital interventions.

The exploration of persuasive design holds promise for enhancing user engagement and improving intervention outcomes (Fogg 1998; Kelders et al., 2012; Orji & Moffatt, 2018). The concept of persuasive design was developed to leverage technology to positively influence behaviour change and discourage harmful behaviour at an individual level (Fogg 1998). This method is based on the concept that technology can serve more than just functional purposes; it can also act as a catalyst to promote and support targeted behaviours, emotions and mental states (Fogg 1998). Through strategic application of persuasive design principles and strategies, these systems aim to motivate and support users in fostering positive shifts in attitude and behaviour (Fogg et al., 2007; Oinas-Kukkonen and Harjumaa, 2008; Orji & Moffatt, 2018).

The Persuasive Systems Design (PSD) framework, proposed by Oinas-Kukkonen and Harjumaa (2008), consists of 28 principles categorised into four domains: 1) Primary Task Support: facilitating the primary goal of the app, 2) Dialogue Support: enabling communication between the app and the user, 3) System Credibility Support: enhancing the trustworthiness and credibility of the app, and 4) Social Support: leveraging a social experience within the app. Each domain is further broken down into 7 distinct persuasive principles. These principles serve as a roadmap for developers to craft persuasive systems that produce more compelling products to engage users and foster positive behaviour change overtime (Parada et al., 2020). See Table 1 for more information.

Table 1

*Persuasive Systems Design Framework adapted from* Oinas-Kukkonen & Harjumaa (2008)
Primary Task Support: Facilitating the completion of the primary tasks or goals that the user aims to achieve through the system	Dialogue Support: Enabling effective communication and interaction between the system and the user	Credibility Support: Enhancing the perceived trustworthiness and credibility of the system	Social Support: Incorporating social influence and social interaction features within the system
Reduction: Reducing complex behaviours into manageable tasks	Praise: Providing praise or positive feedback for using the system	Trustworthiness: Information provided should be fair and objective	Social learning: Providing a way for users to observe the outcomes of others performing target behaviours
Tunnelling: Guiding the user through a process or experience in a manageable way	Rewards: Providing rewards to users for using the system/achieving target behaviours	Expertise: Information provided should demonstrate knowledge, experience, and competence	Social comparison: Providing means for users to compare their performance with other users
Tailoring: Customising information to reflect the needs, interest, personality and usage context of a user group	Reminders: remind users of their target behaviour	Surface credibility: The system conveys a competent appearance	Normative influence: Facilitate the gathering of users with similar goals, fostering a sense of community & shared norms
Personalisation: providing individualised content and experiences	Suggestion: Offer relevant suggestions to the user	Real world feel: Highlight the people or organisation behind the system	Social facilitation: Users can perceive that others on the system are performing the target behaviour
Self-monitoring: Keeping track of a user’s performance or status	Similarity: The system reflects the user or user group	Authority: The system should reference people in roles of authority	Cooperation: Users can co-operate to meet a common goal
Simulation: Providing opportunities to engage in an experience allows users to directly observe the immediate cause-and-effect relationships of the target behaviour	Liking: A visually attractive system	Third Party Endorsements: The system should reference endorsements from respected sources	Competition: Users can compete against other users
Rehearsal: providing opportunities to practice a behaviour to help users prepare for real-world situations	Social Role: System assumes a social, personable, or human like manner	Verifiability: Providing links to third party sources to verify the accuracy of the system’s content	Recognition: Users who perform target behaviours should be recognised publicly on the system

Limited work has examined the role of PSD principles in digital interventions for mental health. Kelders et al. (2012) found a significant relationship between adherence to web-based digital mental health platforms and the application of persuasive design domains. Their findings highlighted that dialogue support domain principles, which enhance effective communication and interaction between the system and the user, were associated with increased adherence to the web-based intervention. This study was limited by the exclusion of interventions due to insufficient usage data and the variability in coding, which was based solely on published descriptions. These factors may have introduced bias and affected the comparability of findings. The authors recommend that future research explore the relationship between persuasive technology, particularly primary task support, and the clinical outcomes of digital interventions.

Orji and Moffatt (2018) reviewed 16 years of literature on persuasive technology in health and wellness. Authors reported that although 92% of the studies reported positive results, the review did not identify a statistically significant correlation between the use of persuasive principles and the intervention outcomes. In 2021, McCall et al. conducted a systematic review, meta-analysis, and meta-regression to examine the impact of persuasive design principles in self-directed eHealth interventions. Their findings provided modest preliminary support that interventions utilizing more persuasive elements in the primary task domain were more effective for treating depression, but not anxiety. In contrast, Wu et al. (2021), in a separate meta-analysis of persuasive design in smartphone apps for anxiety and depression, found a negative association between persuasive design principles and engagement, as measured by completion rates, further highlighting the inconsistent evidence surrounding the role of persuasive design in digital interventions.

Given the rapid proliferation of mental health apps for a range of conditions and the challenge of maintaining user engagement, reviewing the employment of persuasive design principles, and their impact on user engagement rates and the overall effectiveness is both relevant and timely. Previous research has often focused on broader eHealth interventions beyond mental health, restricted its scope to web-based interventions, or concentrated on specific conditions like depression and anxiety. In contrast, this review takes a more targeted approach by examining smartphone apps designed to address all mental health conditions. This focus allows us to fill gaps in earlier studies and provides a more comprehensive understanding of how persuasive design principles influence both engagement and efficacy across diverse mental health domains. Specifically, this review aims to (1) conduct a systematic review and meta-analysis of randomised controlled trials of smartphone mental health apps, (2) conduct a systematic analysis to assess the prevalence and type of persuasive design principles used in these interventions, and (3) via meta-regression examine the relationship between persuasive design principles and efficacy and engagement levels of digital mental health apps. Comprehensively exploring these factors will provide valuable insights to inform the development and improvement of mental health apps, ultimately contributing to enhanced mental health outcomes for users.

This study comprised a systematic review, meta-analyses, and meta-regression, conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines (Page et al., 2021) (Appendix A). Additionally, it adhered to the prospectively registered protocol in PROSPERO (CRD42022352123) on August 4, 2022, which can be accessed at https://www.crd.york.ac.uk/prospero/.

Eligibility Criteria

To be included, studies had to meet the following criteria: (1) randomised controlled trials, (2) delivering a digital mental health intervention, (3) delivered via smartphone app, (4) aimed at addressing a mental health condition with psychological/psychosocial approaches, (5) delivered the therapeutic intervention primarily via the app, (6) reported quantifiable measurements of user engagement, intervention adherence, or effectiveness data, (7) contained an app description that was adequate enough to allow coding of its persuasive design principles, (8) were published in peer-reviewed academic journals and (9) written in English.

Exclusion criteria were: (1) Case studies and feasibility studies without a control group, reviews, theses, and book chapters; (2) data reported in another study (i.e. conference abstracts where data is subsequently published elsewhere).

Synthesis Approach

To examine our primary aims, we conducted a systematic narrative synthesis and meta-analysis of the available literature. The available studies that were identified as meeting criteria via review were systematically synthesised and reported by examining (a) the types of samples, interventions, and mental health contexts in which smartphone-based therapeutic interventions have been applied, (b) which persuasive principles were present in each app description and the number of overall principles present in each app, and (c) the type of metrics used to report on intervention engagement and level of engagement (if available), and d) pre and post intervention effectiveness data.

Search Strategy

A systematic search was conducted using the Ovid platform to identify relevant studies. The databases searched included PsycINFO, PubMed, and Embase, employing search terms designed to target four key concepts: (1) app (app OR smartphone OR iphone OR android OR mobile OR “mobile phone” OR “mobile app” OR “mobile application” OR “mobile based” “smartphone app” OR “smartphone application”), (2) digital intervention (digital intervention” OR “mobile intervention” OR “smartphone intervention” OR “digital mental health” OR “mobile health intervention” OR “digital technology” OR “mHealth” OR “eHealth” AND “mental health”), (3) mental ill health (“mental ill health” OR disorder OR “psychiatric disorder” OR well-being OR wellbeing OR depress* OR psycho* OR bipolar OR anxiety OR schizophrenia OR affective OR self-harm OR self-injury OR distress OR mood OR body image OR eating disorder OR suicid* OR “posttraumatic stress” OR PTSD OR agoraphobia OR phobia* OR panic OR funct* OR OCD OR stress OR symptom,* (4) randomised controlled trial (randomised OR randomized OR RCT OR waitlist OR allocate*). NOT protocol OR “single arm” OR “systematic review” OR “scoping review.”

Study Selection & Data Collection

The searches were completed in August 2022, after which all retrieved articles identified by the search strategy were downloaded and then uploaded to Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org. After automated identification and removal of duplicates, two reviewers (LV, JH) independently screened the title and abstract of all articles according to the eligibility criteria. Once consensus on eligibility was reached in the screening phase, two researchers (LV, JH) independently conducted full-text reviews. If conflicts arose in the screening or full-text review phases via the Covidence interrater feature, the reviewers (LV, JH) resolved them through discussion.

Data Extraction

Four authors (LV, MSV, SO, RS) independently extracted data related to study, sample, intervention characteristics and engagement and efficacy rates (i-v below). 50% of the full data set (119) was double extracted to ensure reliability (WP, RS, LV). Any discrepancies were resolved by consensus with a third author (JN). The following data was extracted:

Study characteristics: authors, year of publication, country of participant recruitment,

Sample characteristics: total sample size, number of participants in control and intervention group(s), primary mental health condition, age, gender, population type (clinical/sub-clinical/non-clinical),

Intervention characteristics: app name, description, theoretical treatment model (e.g., CBT), intervention length (in days), if there was human involvement, control conditions (e.g., waitlist, treatment as usual, active control), primary mental health target including discrete symptoms (i.e. stress), symptoms clusters (i.e. a depression diagnoses) and measures of behaviour (i.e. social functioning),

Engagement rates: whether engagement data were reported (yes/no), engagement/adherence metric (e.g., number of logins, modules completed), and engagement rates,

Efficacy data: means, standard deviations (SDs), and sample sizes (N) of the primary outcome measure for both intervention and control group(s) at pre and post-intervention time points. If certain metrics (e.g., SDs) were missing, they were either calculated using available data or authors were contacted for the additional information. When multiple primary outcome measures were presented, the first listed measure was considered the primary outcome for extraction. An exception to this procedure was made for subsequent outcome measures pertaining to depression or anxiety, which were also included in the extraction process, and

Persuasive principles: the presence or absence of the 28 persuasive design principles outlined in the Persuasive Systems Design (PSD) framework for each app description. Each principle was coded dichotomously for each app, with a value of 1 indicating its presence and 0 indicating its absence, resulting in a total out of 28. Data was extracted from description of app within the included studies, and, whenever possible, app information from supplementary materials such as protocols, development papers, app stores, or websites were gathered and used to provide more context for each app.

These data were extracted by three authors (KB, LB, LV), who first collaborated to operationalise the PSD framework (Table 1.) and apply its principles to contemporary app design. KB, a senior UX designer, and LB, a senior product designer, are both experts in product design and user needs. Initially, the three authors coded 20% of the apps collaboratively to ensure consistency in their approach before coding independently once they were confident that they were aligned in their deductive approach. This collaborative effort aimed to maintain accuracy and consistency in applying the PSD framework to the app descriptions. In case of discrepancies requiring resolution, a fourth author (JN) was consulted. During this coding process, reoccurring principles that were not part of the PSD framework were identified and discussed among the reviewers. These principles were then added to the deductive coding framework for later analysis.

Risk of Bias assessment of individual studies

Six reviewers independently evaluated the risk of bias in each study using the Cochrane Risk of Bias (RoB2) tool. Each of the five risk domains of the RoB2 were scored against a three-point rating scale relating to a low, some concern, or high risk of bias.

Meta-Analysis, Meta-Regression, and Correlation Analysis

A meta-regression analysis was conducted to investigate the relationship between intervention efficacy and the implementation of persuasive design principles. Additionally, this analysis examined the association between intervention efficacy and variables related to sample type (e.g., clinical, sub-threshold, non-clinical), outcome type (e.g., depression, anxiety etc), the presence of human support (blended vs self-guided), and intervention type (e.g., CBT, mindfulness etc.) A two-tailed Pearson’s correlation analysis was also performed to examine the relationship between the number of persuasive design principles and app engagement. All analyses were conducted in SPSS (v29).

The meta-analysis and meta-regression used random-effects models with maximum likelihood estimations. Initially, we computed Hedges' g effect sizes from available estimates (e.g., M, SD) reported within studies to reflect the size of the between-group (control vs. intervention) effects at the post-intervention time-point, and then meta-analysed them. The primary outcome variable (e.g., depression) as reported within each study was used across all analyses. To maintain coding consistency, estimates calculated based on positive-valued outcomes (i.e., where higher scores indicate better mental health) were reverse-coded to align with the majority of outcomes. Therefore, negative Hedges' g values represent cases where the intervention group was more effective than the control group. To gain a deeper understanding of intervention efficacy, we conducted sub-group moderator analyses across various variables, including sample type (non-clinical, sub-clinical, clinical), outcome type (depression, anxiety, stress, social anxiety, PTSD, body image/eating disorder, psychosis, suicide/self-harm, general mental health, postnatal depression), presence of human support (blended vs. self-guided), and type of intervention (CBT, CBT+, mindfulness-based, psychoeducation). This approach allowed us to identify specific factors that may influence the overall effectiveness of digital mental health interventions.

Additionally, a meta-regression was performed, regressing the efficacy of the interventions (Hedges' G) on the total number of persuasive design principles identified in each study, to determine the potential relationship between persuasive design principles and intervention efficacy. Throughout all analyses, we assessed heterogeneity using common metrics (e.g., Cochran's Q, I²; Cochran, 1954; Higgins et al., 2003) and examined potential publication bias by (a) visually inspecting the funnel plot of standard errors by Hedges' G to assess for asymmetry, and (b) examining trim and fill estimates to determine if the potential for missing estimates affected the overall effect size (Duval & Tweedie, 2000).

The two-tailed Pearson’s correlation analysis was run to examine the relationship between number of persuasive design principles used and app engagement, as there have been conflicting findings in the literature conveying how persuasive design principles are associated with levels of engagement.

Included studies

The initial search yielded 5,027 articles, which were screened for duplicates, resulting in 4,025 articles eligible for title and abstract review. From this, 390 articles advanced to full-text review. Following this phase, 273 articles were excluded, leaving 119 that met the predefined inclusion criteria for the systematic review. Of these, 92 of provided sufficient data for inclusion in the meta-analysis. The remaining 27 studies could not be included in the meta-analysis due to limitations such as inadequate pre/post-outcome data or insufficient intervention descriptions but were included in the overall synthesis. The PRISMA 2020 flow diagram illustrating the article selection process is presented in Fig. 1.

The two title and abstract screeners (LV and JH) had an 89.81% proportional agreement (3615/4025) for retaining or removing articles during the title screening, and a 77.24% agreement (275/356) during the full-text screening.

Study Characteristics

A total of 119 studies were included in the systematic review, encompassing a total of 30,251 participants. These studies were published between 2016 and 2022 and involved participants from 27 different countries. The United States represented the largest proportion of participants at 31.1%, followed by the United Kingdom and Germany at 8.4% each, and Australia at 7.6%.

Non-clinical samples constituted 30.3% (n = 36) of the studies, sub-clinical samples 40.3% (n = 48), and clinical samples 29.4% (n = 35). The mean participant ages ranged from 14 to 60 years old (median = 34 years). The majority (67%; n = 80) of the interventions were entirely self-guided, with the remaining 33% (n = 39) being blended, integrating some form of human support to the intervention. Notably, apps were more likely to include human support in clinical (60%: 21/35) and sub-clinical (29%: 14/48) samples than in non-clinical samples (11%: 4/36). Intervention durations ranged from 10 days to 18 months.

Depression was the most targeted mental health condition, accounting for 25.64% of the studies. Studies that specifically targeted a combination of depression, anxiety, and stress constituted 11%, while stress alone was the focus of 9.4% of studies. Other mental health conditions targeted include eating disorders and body dissatisfaction/dysmorphic disorder (11%), generalised anxiety (7.69%), post-traumatic stress disorder (PTSD; 5.98%), general mental health and wellness (4.27%), psychotic spectrum disorders (3.4%), postnatal depression (3.4%), psychological distress (3.4%), a combination of psychotic spectrum and bipolar disorder (2.56%), non-suicidal and suicidal injury (1.7%), obsessive-compulsive disorders (1.7%), agoraphobia and panic disorder (1.7%), suicidal ideation (0.85%), sleep disturbance (0.85%), resilience (0.85%), loneliness (0.85%), burnout (0.85%), and bipolar disorder (0.85%).

In the reviewed studies, most interventions were grounded in one or more psychological theories. Cognitive-Behavioural Therapy (CBT) was the most common, featured in 60.5% of interventions (n = 72). Mindfulness was used in 23.5% (n = 28), Acceptance and Commitment Therapy (ACT) in 4.2% (n = 5), and Dialectical Behaviour Therapy (DBT) in 2.5% (n = 3). A small proportion of studies (5%, n = 6) did not specify any therapeutic framework. The characteristics of each study are presented in Table 2.

Table 2

*Characteristics of Included Studies*
Study Number	Authors, Year, Country of Recruitment	App	No. of PSD	Mental Health Target	Therapeutic Modality	Study n	Length of Intervention (Days)	Age (Sample Mean)	Blended?	Engagement Reported?	Sample	Risk of Bias²
1	Akin-Sari et al., 2022, Turkey	GG OCD	4	OCD	CBT	55	12		No	No	Sub-Clinical	++++?
2	Al-Refae et al., 2021, Canada	Serene	5	Depression, Anxiety, Stress	Mindfulness	245	28	25.24	No	Yes*	Non-Clinical	+++?+
3	Alvarez-Jimenez et al., 2021, Australia	Horyzons	10	First-Episode Psychosis	CBT	170	548	20.91	Yes	Yes	Clinical	+++++
4	Andersson et al., 2021, Sweden	The Compassion Mindset The Mindfulness App	4 3	Stress	Neuroscience, CBT psychodynamic therapy, affect theory, attachment theory, mindfulness, loving kindness meditation	57	42	25	No	Yes+	Non-Clinical	+++??
5	Araya et al., 2021, Brazil & Peru	CONEMO (Emotional Control)	x	Depression	Behavioural Activation	1312	91		Yes	Yes	Clinical	++++?
6	Arean et al., 2016, US	Project: EVO iPST	x x	Depression	Modulate Cognitive Control /Problem Solving therapy	626	28	33.9	No	Yes	Sub-Clinical	+++?+
7	Bakker et al., 2018, Australia	MoodKit MoodPrism MoodMissio	4 3 4	Depression	CBT	312	30	34.2	No	No	Non-Clinical	?+++?
8	Ben-Zeev et al., 2018, US	FOCUS	5	Serious Mental Illness		163	91	49	Yes	Yes	Clinical	++++?
9	Ben-Zeev et al., 2021, US	CORE	6	Serious Mental Health Problems	To challenge dysfunctional thoughts that underlie common symptoms of SMI, self-stigmatizing attitudes, and maladaptive beliefs that impede treatment seeking and recover	315	30		No	Yes*	Clinical	?+++?
10	Birney et al., 2016, Australia	MoodHacker	x	Depression	CBT	300	30	40.7	No	Yes	Sub-Clinical	+++???
11	Boettcher et al, 2018, Sweden	Challenger	12	Social Anxiety	CBT	209	42	35.4	No	Yes*	Clinical	+?++?
12	Børøsund et al., 2020, Northern Europe	StressProffen	11	Stress	CBT	175	91	52	Yes	Yes	Non-Clinical	++++?
13	Bruehlman-Senecal et al., 2020, US	Nod	11	Loneliness	CBT	221	28	18.7	No	Yes	Non-Clinical	++++?
14	Bruhns et al., 2021, Germany	MTC & More	x	Depression	CBT, Mindfulness, Acceptance & Commitment Therapy, & Metacognitive Training	423	28	22.98	No	Yes	Sub-Clinical	+?++?
15	Bucci et al., 2018, UK	Actissist ClinTouch	7 3	Psychosis	CBT	36	84		No	Yes	Clinical	+++++
16	Bush et al.,2017, US	Virtual Hope Box	3	Stress	CBT	118	84		Yes	Yes*	Sub-Clinical	++++?
17	Carl et al., 2020, UK	Daylight	9	GAD	CBT	256	42	30.9	No	Yes	Clinical	+++?+
18	Catuara-Solarz et al., 2022, UK	Foundations	6	Anxiety & Stress	CBT, ACT, positive psychology, & sleep science	190	28		No	Yes	Sub-Clinical	+-+??-
19	Cerea et al., 2020, Italy	GG Relationship Doubts (GGRO)	7	Relationship Obsessive Compulsive Disorder (ROCD)	CBT	50	15	22	No	No	Sub-Clinical	+++??
20	Cerea et al., 2021 Italy	GGBI: Positive Body Image	6	Body dissatisfaction and BDD/ED symptoms	CBT	50	16	21.74	No	Yes	Sub-Clinical	+++??
21	Cerea et al., 2022, Italy	GGBI	3	Body dissatisfaction and BDD/ED symptoms	CBT	95	16	22.82	No	No	Sub-Clinical	+++??
22	Chan et al., 2019, Hong Kong	iParent	5	Postnatal Depression	Psychoeducation	660			Yes	No	Non-Clinical	++++?
23	Chan et al., 2021, UK	proACT-s	4	Depression & Insomnia	CBT	320	42	27.3	No	No	Clinical	++++?
24	Christoforou et al., 2017, UK	Agoraphobia Free	8	Agoraphobic and panic symptoms	CBT	142	84	39.7	No	No	Sub-Clinical	++++?
25	Chu et al., 2022, US	Headspace	9	Stress & Burnout	Mindfulness	87	70		No	Yes^	Non-Clinical	+++++
26	Cuijpers et al., 2022, Lebanon	Step-by-Step	8	Depression	Psychoeducation, Behavioural activation, Illustrative narrative	569	56	27.02	Yes	Yes	Clinical	+++++
27	Cuijpers et al., 2022, UK	Step-by-Step	8	Depression	Psychoeducation & Behavioural Activation	680	56	31.5	Yes	Yes	Clinical	+++++
28	Dagoo et al., 2014, Sweden	No Name	5	Social Anxiety Disorder	CBT	52	63	36.81	No	Yes	Clinical	?+++?
29	Dahne et al, 2019, US	¡Aptívate!	6	Depression	Behavioural activation	42	56	36.05	No	Yes*	Sub-Clinical	-+++?
30	Deady et al., 2020, Australia	HeadGear	7	Depression	Behavioural Activation & Mindfulness	2275	35	40.26	No	Yes	Sub-Clinical	+++++
31	DeKock et al., 2022, UK	NHSWBP Staff Wellbeing Project My Possible Self	7 3	Depression, Anxiety, & Mental Wellbeing	CBT, Problem Solving Therapy, Interpersonal Psychotherapy, Positive Psychology	169	28		No	Yes	Non-Clinical	++++?
32	Depp et al., 2015, US	Personalized Real-Time Intervention for Stabilizing Mood (PRISM)	5	Bipolar Disorder I or II	Pyschoeducation & Self-monitoring	104	70	47.5	Yes	No	Clinical	++++?
33	Depp et al., 2019, US	CBT2go	4	Schizophrenia or Bipolar Disorder	CBT	255	84		Yes	Yes	Clinical	+++?+
34	DiSimplico et al., 2020, UK	Imaginator	5	Self-Harm Behaviour	Functional Imagery Training	38	91		Yes	Yes	Sub-Clinical	++?+?
35	Ebenfield et al., 2021, Germany	GET.ON Panic	7	Panic Disorder	CBT	92	56	38.36	Yes	Yes	Sub-Clinical	+++++
36	Ebert et al., 2018, Germany	GET.ON Mood Enhancer	8	Major Depressive Disorder	Behavior therapy (BT) and problem-solving therapy (PST)	204	49	44.2	Yes	Yes	Sub-Clinical	+??+?
37	Ebert et al., 2021, Germany	GET.ON Stress	8	Occupational Stress	Lazarus’ transactional model of stress	396	49	41.76	No	Yes	Non-Clinical	++++?
38	Economides et al., 2018, US	Headspace	9	Stress	Mindfulness	160	10		No	Yes	Non-Clinical	++?+?
39	Enock et al., 2014, US	No Name	1	Social Anxiety	Attention Bias Modification Training	429	28	34.8	No	Yes	Non-Clinical	???+?
40	Everitt et al., 2021, Australia	Mood Tracker ImproveYourMood ImproveYourMood+	x x x	Depressive Symptoms	CBT, Mindfulness	235	21	32.97	No	Yes	Non-Clinical	?+?+?
41	Ferré-Grau et al., 2021, Spain	TIVA	8	Burnout	Positive Psychology	131	28	60.65	Yes	No	Non-Clinical	+++++
42	Flett et al., 2019, New Zealand	Headspace Smiling Mind	9 7	Depressive Symptoms, Anxiety, & Stress	Mindfulness Meditation	210	30	20.8	No	Yes	Non-Clinical	?++++
43	Flett et al., 2020, New Zealand	Headspace	9	Psychological Distress	Mindfulness Meditation	250	91	17.87	No	Yes	Non-Clinical	?++++
44	Franklin et al., 2016, US	Therapeutic Evaluative Conditioning (TEC)	x	Non-suicidal & suicidal self-injury			28			Yes	Sub-Clinical	???+?
45	Fuller-Tyszkiewicz et al., 2020, Australia	StressLess	6	Depression, Anxiety, & Stress	Third wave (mindfulness-based) CBT	183	35	39.5	No	Yes	Non-Clinical	?+++?
46	Gao et al., 2022, US	Unwinding Anxiety	5	Sleep Disturbance	Mindfulness Training	80	119		No	Yes	Sub-Clinical	+?+++
47	Ghanbari et al., 2021, Iran	The Breast Cancer Support Zone	2	Anxiety & Self-Esteem	Psychoeducation	82	28	46.45	No	No	Non-Clinical	+?+??
48	Gnanapragasam et al., 2022, UK	Foundations	x	General (non-psychotic) Psychiatric Morbidity	CBT, mindfulness-based CBT, relaxation techniques and positive psychology	1002	56	44.3	No	Yes	Non-Clinical	++?++
49	Graham et al., 2020, US	Intellicare	5	Depression & Anxiety		146		42.3	Yes	Yes	Sub-Clinical	+?+++
50	Greer et al., 2019, US	No Name	x	Anxiety	CBT	145	84	56.45	No	Yes	Clinical	++++?
51	Ham et al., 2019, South Korea	HARUToday HARUCard	9 4	Depression & Anxiety	CBT	80	66		No	No	Sub-Clinical	++++?
52	Harrer et al., 2021, Germany	StudiCare Fernstudierende	6	Depression	Problem-oriented coping, & emotion-oriented coping through emotion regulation strategies	200	49	36.97	Yes	Yes	Sub-Clinical	++??+
53	Heim et al., 2021, Lebanon	Step-By-Step	4	Depression, functional impairment, anxiety, PTSD & wellbeing	CBT	138	35		Yes	Yes	Sub-Clinical	+??++
54	Hensler et al., 2022, Sweden	PTSD Coach	5	Posttraumatic stress, depressive, & somatic symptoms	CBT	179	91	42.78	No	Yes*	Sub-Clinical	+?++?
55	Hicks et al., 2021, US	Headspace	9	Anxiety	Mindful Meditation	169	14		No	No	Non-Clinical	+????
56	Hides et al., 2019, Australia	Music eScape	1	Distress	The dynamic information-motivation-behavioural skills health behaviour model	66	168	19.9	No	Yes	Sub-Clinical	+?++?
57	Hildebrandt et al., 2017, US	Noom Monitor	1	Binge-eating disorder or bulimia nervosa	CBT	109	84	32.11	Yes	No	Clinical	-+++?
58	Hirshberg et al., 2022, US	Healthy Minds Program	5	Psychological Distress	Awareness, connection, insight, and purpose model of well-being	239	28		No	Yes	Non-Clinical	-??+?
59	Huberty et al., 2019, US	Calm	9	Stress	Mindfulness	109	56		No	Yes	Non-Clinical	+?+++
60	Huberty et al., 2021, US	Calm	9	Depression & Anxiety	Mindfulness	239	56	44.5	No	Yes	Sub-Clinical	-?++?
61	Hur et al., 2018, US	Todac Todac	8	Depression	CBT	34	21	23.71	No	No	Clinical	+??+?
62	Hwang et al., 2019, South Korea	No Name	3	Stress		60	30		No	No	Non-Clinical	+?++?
63	Hwang et al., 2022, South Korea	BetterLife	7	Occupational Stress	CBT	126	70		No	No	Sub-Clinical	+??+?
64	Ivanova et al., 2016, Sweden	Ångesthjälpen (The Anxiety Help)	8	Social Anxiety Disorder and Panic Disorder	ACT	152	70	35.92	Yes	Yes	Clinical	+?+++
65	Jannati et al., 2020, Iran	Happy Mom	6	Postpartum depression	CBT	75	61	27	No	No	Clinical	??++?
66	Kageyama et al., 2021, Japan	The Subliminal Priming with Supraliminal Reward Stimulation (SPSRS)	4	Depression	Verbal stimulation from research-based evidence & positive language	32	35		No	Yes	Sub-Clinical	+?+++
67	Kahn et al., 2016, US	Mission Reconnect (MR)	5	Mental Health & Wellness	Mindfulness-based therapies, massage therapy, positive emotions, and caregiver education	180	112		No	Yes	Non-Clinical	+?++?
68	Kollei et al., 2017, Germany	Mindtastic Body Dissatisfaction app (MT-BD)	x	Body Dissatisfaction	Gamification strategies to systematically foster approach of functional and avoidance of dysfunctional stimuli	53	14		Yes	No	Sub-Clinical	?++??
69	Kranenburg, et al., 2022, Netherlands	No Name	6	Stress	Mindfulness	811	56	45.86	No	No	Non-Clinical	+?-+?
70	Kruzan et al., 2022, US, UK, EU	TalkLife	x	Non-suicidal self-injury	Peer Support	220	56	20.32	No	No	Sub-Clinical	+???+
71	Kubo et al., 2019, US	Headspace	9	Stress & Quality of Life	Mindfulness	97	56		No	Yes^	Non-Clinical	+??++
72	Kuhn et al., 2017, US	PTSD Coach	5	PTSD	CBT & Mindfulness	120	91	39	No	Yes*	Clinical	++???
73	Laird et al., 2022, US	Calm	9	Stress	Mindfulness Based Meditation	74	28		No	Yes	Sub-Clinical	+??+?
74	Lee & Jung., 2018, Canada	DeStressify	3	Stress, Anxiety, Depression	Mindfulness	206	28		No	Yes*	Non-Clinical	+??+?
75	Levin et al., 2022, US	Stop, Breathe, and Think	7	General Psychopathology	Mindfulness	23	28	20.43	No	Yes*	Non-Clinical	+??+?
76	Linardon et al., 2022, Australia	Break Binge Eating	7	Binge eating	CBT	392	28		No	Yes	Sub-Clinical	++-+?
77	Linardon et al., 2022, Australia	Break Binge Eating Breaking the Diet Cycle	7 5	Binge Eating	CBT	600	28	28.95	No	Yes	Sub-Clinical	++-+?
78	Litvin et al., 2020, UK	eQuoo	6	Resilience	CBT	709	35		No	Yes*	Non-Clinical	++?++
79	Lukas & Berking., 2021, Germany	Mentalis Phoenix (MT-Phoenix)	6	Depression	Avoidance Modification Training	77	14		No	Yes	Sub-Clinical	?????
80	Lukas et al., 2021, Germany	Mentalis Phoenix (MT-Phoenix)	6	Depression	CBT with gamified approach-avoidance bias modification training	30	14	29.31	Yes	Yes	Sub-Clinical	+++??
81	Ly et al., 2014, Sweden	No Name No Name	7 7	Depression	Behavioural Activation	84	56	36	Yes	No	Clinical	++++?
82	Ly et al., 2015, Sweden	No Name	7	Depression	CBT & Behavioural Activation	93	63	30.6	Yes	No	Clinical	++?+?
83	Mak et al., 2018, China	Living With Heart (LWH)	x	Mental Wellbeing	Mindfulness, Self-compassion, CBT	2282	28		No	Yes	Non-Clinical	+++++
84	Mantani et al., 2018, Japan	Kokoro-app	10	Depression	CBT - self-monitoring, behavioural activation, & cognitive restructuring	164	63		No	Yes	Clinical	+++??
85	McCloud et al., 2020, UK	Feel Stress Free	10	Depression & Anxiety	CBT	168	42	24.3	No	Yes*	Sub-Clinical	+++??
86	Meinlschmidt et al., 2016, South Korea	No Name	x	Improving Mood	Viscerosensory attention, emotional imagery, facial expression, & contemplative repetition	31	13	24.32	Yes	Yes	Non-Clinical	?++++
87	Miller-Graff et al., 2021, Egypt	PTSD Coach	5	PTSD	CBT	87	91		No	Yes*	Sub-Clinical	+??+?
88	Miner et al., 2016, US	PTSD Coach	5	PTSD	CBT	49	28	45.7	No	Yes*	Sub-Clinical	+++??
89	Moberg et al., 2019, US	Pacifica	9	Stress, Anxiety, & Depression	CBT and mindfulness	500	30		No	Yes	Sub-Clinical	+++??
90	Neumayr et al., 2019, Germany	Recovery Record	x	Anorexia Nervosa	CBT with additional elements of DBT, ACT, & MET.	40	56		Yes	Yes	Clinical	+????
91	Newman et al., 2021, US	No Name	7	Generalised Anxiety Disorder	CBT	100	91		Yes	No	Clinical	++???
92	O'Dea et al., 2020, Australia	WeClick	x	Depression	CBT, Help-seeking intentions, Social Learning, Skill Development	193	28	14.82	No	Yes	Non-Clinical	+?++?
93	O'Toole et al., 2019, Denmark	LifeApp’tite	x	Suicide Risk & Depression	Psychoeducation	154	56		Yes	Yes	Sub-Clinical	++???
94	Orosa-Duarte et al., 2021, Spain	REM Volver a casa ‘Mindfulness-Based Emotion Regulation. Going Home’	5	Anxiety	Mindfulness	129	56		No	No	Non-Clinical	++?+?
95	Pacella-LaBarbara et al., 2020, US	PTSD Coach	5	Posttraumatic Stress Disorder (PTSD)	CBT	64	30	37	No	Yes*	Non-Clinical	+????
96	Pensak et al., 2021, US	Pep-Pal	5	Depression, Anxiety, & Stress	CBT	56	84	54.3	No	Yes	Clinical	+??++
97	Raevuori et al., 2021, US	Meru Health Program (MHP)	6	Depression	Mindfulness-Based Stress Reduction, Mindfulness-Based Cognitive Therapy, CBT, & Behavioural Activation	124	56	25.1	Yes	Yes	Clinical	+++??
98	Rodante et al., 2022, Argentina	CALMA	7	Suicidal Behaviour	DBT	56	28		Yes	No	Sub-Clinical	-????
99	Roepke et al., 2015, US	SuperBetter – CPT-PTT SB General – SuperBetter	8 x	Depression	CBT & positive psychotherapy	283	30	40.15	No	No	Clinical	++???
100	Rohr et al., 2021, Germany	Sanadak	6	Posttraumatic Stress	CBT	133	28		No	No	Sub-Clinical	+++??
101	Roncero et al., 2019, Israel	GGRO	x	OCD	CBT	97	15		No	Yes	Non-Clinical
102	Roy et al., 2021, US	Unwinding Anxiety	7	Anxiety	Mindfulness	65	61		No	Yes	Sub-Clinical	+++??
103	Sawyer et al., 2019, Australia	eMums Plus	8	Post-Natal Depression		133	112		Yes	Yes^	Sub-Clinical	+?+++
104	Schlicker et al., 2020, Germany	Get.Back	x	Depression	CBT	76	63	50.78	Yes	Yes	Clinical	+?+??
105	Schlosser et al., 2018, US	PRIME	x	Schizophrenia	CBT	43	84		Yes	Yes	Clinical	?++??
106	Stallman et al., 2019, Australia	My Coping Plan	4	Psychological distress	Strengths-focused coping planning approach	56	30	28.79	No	Yes	Sub-Clinical	+?++?
107	Steare et al., 2020, UK	My Journey 3	6	First-Episode Psychosis		40	356		Yes	Yes	Clinical	+++++
108	Stiles-Shields et al., 2019, US	Boost Me Thought Challenger	5 5	Depression	Behavioural Activation/ Cognitive Therapy	30	42		Yes	Yes^	Clinical	+++++
109	Stolz et al., 2018, Switzerland	No Name	7	Social Anxiety Disorder	CBT	150	84		Yes	Yes	Clinical	+++++
110	Sun et al., 2021, China	Spirits Healing (in Chinese)	8	Maternal Perinatal Depression	Mindfulness	168	56	29.91	No	Yes	Sub-Clinical	+++++
111	Tonning et al., 2021, Denmark	The Monsenso system	x	Unipolar Depressive Disorder	CBT	120	183		Yes	Yes	Clinical	+?+++
112	Torok et al., 2022, Australia	LifeBuoy	6	Suicidal Ideation	DBT	455	42	21.5	No	Yes	Non-Clinical	+++++
113	Tregarthen et al., 2019, US	Recovery Record	x	Eating Disorders	CBT-based eating & symptom monitoring	3294	56		No	No	Clinical	++?+?
114	van der Meer et al., 2020, Netherlands	SUPPORT Coach	x	PTSD	CBT	287	28	43.37	No	Yes	Sub-Clinical	+??++
115	vanAubel et al., 2020, Belgium	PsyMate	5	Depression & Psychotic Symptoms	ACT	55	42		Yes	Yes	Sub-Clinical	+?+++
116	Wilhelm et al., 2022, US	Perspectives	5	Body Dysmorphic Disorder	CBT	80	84		Yes	Yes*	Clinical	+++++
117	Wong et al., 2021, Hong Kong	Lifestyle Hub	6	Depression	Guided by the theoretical constructs of the transtheoretical model (TTM), including consciousness-raising, dramatic relief, counterconditioning, self-re-evaluation, environmental re-evaluation, self-liberation, contingency management, & stimulus control.	79	56		No	Yes	Sub-Clinical	+++?+
118	Yang et al., 2018, US	Headspace	9	Stress	Mindfulness	88	30		No	Yes	Non-Clinical	+??+?
119	Yang et al., 2022, South Korea	HARU ASD	8	Anxiety	CBT	30	66		No	No	Sub-Clinical	++???

__: Data not reported

*: Self-reported participant engagement data

x: App description not sufficient to code

ROB²: +=low risk, ?=some concerns, -=high risk

Risk of Bias

We assessed for the potential for bias in the included studies using the 5 domains from the Cochrane risk of bias tool version 2 (Sterne et al., 2019). Out of the 119 studies included in the review 7% were determined to have a high risk of bias in at least one domain, while only 13% were identified as having low risk of bias across all assessed domains. The majority of studies (80%) were categorised as having low or unclear risk in each domain. The specific risk of bias for each study is detailed in Table 2.

Publication Bias

Publication bias for the overall effects meta-analysis was explored via visual inspection of the funnel plot of standard errors and trim and fill estimates. As shown in Fig. 4, the funnel plot appears symmetrical across both sides of the final estimate. Further, trim and fill analyses revealed no imputed studies on the left- or right-hand side of this estimate. Hence, we noted limited evidence of publication bias.

1. Efficacy

Meta-Analysis: Overall and Sub-Group Effects of Intervention Efficacy

Of all studies included in this review, 92 (N = 16,782 total participants) provided sufficient information for effect sizes to be calculated for intervention efficacy and were therefore included in the meta-analysis. Across these studies, sample sizes ranged from 16 to 2,271 participants (M = 182.41, SD = 266.53) at study intake. There was an overall significant effect of intervention efficacy, with a medium size (Hedges g = .43, 95% CI [-.53, − .34], p < .001) indicating that at the post-intervention timepoint across studies, those in intervention groups showed significant improvements in their mental health compared with those in control groups. It is worth noting, however, that there was substantial heterogeneity present within this effect (I² = 83.4, range 46.3–98.7). Thus, we examined potential moderators to try and explain some of this variance. Table 3. displays the meta-analytic effects and heterogeneity estimates for both the total effect and sub-group moderation analyses, and Fig. 2. presents the forest plot of effect estimates per study.

The sub-group moderation analysis found the overall effect of intervention efficacy did not significantly differ according to sample types (Q(2) = 3.86, p = .145), outcome types (Q(9) = 14.48, p = .106), the presence of human support in interventions (Q(1) = 0.07, p = .789), or intervention type (Q(4) = 3.40, p = .493) (Table 3).

The analysis identified significant efficacy of digital interventions compared to control conditions when targeting depression, anxiety, stress, PTSD, and body image/eating disorders. No significant effects were observed for social anxiety, psychosis, suicide/self-harm, broad mental health outcomes, or postnatal depression. It is important to note that these latter outcomes were represented by only 2–4 studies each, whereas depression, anxiety, and stress were represented by 8 to 29 studies, providing a more robust pooled effect size.

Table 3

*Meta-Analytic and Heterogeneity Estimates for the Overall Effect, and Sub-Group Moderation Effects, of Intervention Efficacy*
			Effect Size Estimates				Heterogeneity Estimates
		k	Hedge’s g	SE	[95% CI]	p	Cochran’s Q	I²
Overall effect		92	− .43	.05	[-.53, − .34]	< .001	Q(91) = 414.76, p < .001	83.4
Sample type
	Non-clinical	28	− .30	.08	[-.45, − .15]	< .001	Q(27) = 128.28, p < .001	82.8
	Sub-clinical	37	− .49	.07	[-.62, − .36]	< .001	Q(36) = 132.07, p < .001	74.5
	Clinical	27	− .49	.12	[-.72, − .26]	< .001	Q(26) = 138.97, p < .001	87.5
Study outcome
	Depression	29	− .42	.08	[-.57, − .27]	< .001	Q(28) = 113.94, p < .001	79.2
	Anxiety	8	− .71	.15	[-1.01, − .41]	< .001	Q(7) = 20.87, p = .004	70.5
	Stress	21	− .38	.07	[-.52, − .24]	< .001	Q(20) = 59.37, p < .001	67.5
	Social anxiety	4	− .55	.24	[-1.02, − .08]	.021	Q(3) = 19.89, p < .001	81.2
	PTSD	5	− .28	.10	[-.48, − .09]	.005	Q(4) = 4.43, p = .351	16.8
	Body image / Eating disorder	5	− .73	.15	[-1.03, − .43]	< .001	Q(4) = 9.89, p = .042	65.5
	Psychosis	2	− .08	.43	[-.93, .76]	.851	Q(1) = 2.46, p = .117	59.4
	Suicide / Self-harm	3	− .21	.23	[-.66, .24]	.358	Q(2) = 3.68, p = .159	46.3
	Mental health (broadly)	2	.43	.55	[-.64, 1.51]	.433	Q(1) = 18.72, p < .001	94.7
	Postnatal depression	3	-1.08	.92	[-2.88, .72]	.240	Q(2) = 67.85, p < .001	98.7
Human support within intervention
	No	66	− .44	.06	[-.55, − .33]	< .001	Q(65) = 307.33, p < .001	83.8
	Yes	26	− .41	.09	[-.59, − .23]	< .001	Q(25) = 107.40, p < .001	82.7
Intervention type
	CBT (only)	34	− .54	.09	[-.71, − .37]	< .001	Q(33) = 147.62, p < .001	82.7
	CBT (+ others)	11	− .37	.12	[-.62, − .13]	.003	Q(10) = 34.80, p < .001	72.4
	Mindfulness-based	18	− .37	.09	[-.53, − .20]	< .001	Q(17) = 50.66, p < .001	73.8
	Psychoeducation	6	− .53	.14	[-.81, − .24]	< .001	Q(5) = 29.58, p < .001	83.8
	Others	20	− .33	.13	[-.58, − .07]	.012	Q(19) = 112.13, p < .001	88.5
Notes. k = number of studies included.

2. Persuasive Design Principles

None of the studies included in this review explicitly identified the use of specific persuasive design principles. However, 88% (n = 81) of studies included in the meta-analysis provided sufficient description of the app intervention or referenced protocol papers, development papers, websites, or app store information, which allowed for deductive coding based on the PSD framework (Oinas-Kukkonen and Harjumaa, 2009).

A total of 22 out of the 28 persuasive principles in the PSD framework were identified across the reviewed apps. The number of principles per app ranged from 1 to 12, with a mode of 5. Table 4 outlines the specific principles identified in each study, along with updated examples from the apps included in this review. Across the four domains, primary task support, which facilitates the completion of primary tasks or goals that the user seeks to achieve through the system, was the most frequently coded, accounting for 50% of the persuasive principles identified. This was followed by dialogue support (23%) and credibility support (22%). Only 5% of the persuasive principles were derived from the social support domain, which includes principles such as social learning and cooperation. Tunnelling, which guides the user through a process or experience in a manageable way, was the most frequently identified principle, present in 88% of the apps. Rehearsal, which provides opportunities to practice a behaviour to help users prepare for real-world situations, was identified in 84% of the apps. The results are presented below in Table 4.

In addition to the persuasive principles outlined in the PSD framework, this study identified additional recurring design elements within the evaluated apps that were likely persuasive in nature. Identified elements included 1) staged disclosure, 2) goal setting, 3) explicitly self-paced, and 4) limited use. These features are detailed in Table 5 and represent design strategies that may contribute to engagement with digital mental health interventions.

Table 4

*Prevalence and Examples of Persuasive Design Principles Used in Digital Mental Health Apps: Percentage of Studies Reporting Each Principle*
Persuasive Design Principle	Number of studies % (n)	Example of use in app
Tunnelling	88% (81)	"This intervention has modules that cover a variety of topics and can be accessed in any order, including coping effectively with depression and anxiety, enhancing happiness, improving sleep quality, and practicing mindfulness." (DeKock et al., 2022)
Rehearsal	84% (77)	"In each web session, participants were encouraged to practice the prescribed strategies via several homework exercises. These homework exercises were presented in the app component of the intervention, which allowed users to practice these skills digitally and in their daily life." (Linarndon et al., 2022)
Trustworthiness	80% (74)	"The Healthy Minds Program (HMP) smartphone-based meditation intervention was constructed around the awareness, connection, insight, and purpose (ACIP) model of well-being proposed by Dahl and colleagues (2020)." (Hirshberg et al., 2022)
Reminders	55% (51)	"As in the treatment condition, the app emits an alarm prompting participants to access the app at 3 pseudo-randomized time points per day, 6 days a week between 10.00 and 22.00 for 12 weeks alongside usual treatment." (Bucci et al., 2018)
Personalisation	50% (46)	"Guided by the individual formulation, clinicians sent each client weekly tailored content suggestions." (Alvarez-Jimenez et al., 2021)
Self-Monitoring	49% (45)	"Participants rated their anxiety before and after relaxation sessions using a visual analog scale." (Christoforou et al., 2017)
Suggestion	48% (44)	"The main component of the app is a 30-day intervention involving one 5–10 min ‘challenge’ per day." (Deady et al., 2022)
Expertise	43% (40)	"Coaches were 2 bachelor’s degree–level individuals who received training in the coaching manual and weekly supervision with a clinical psychologist." (Graham et al., 2020)
Reduction	32% (29)	"Besides written materials, all contents have also been audio-recorded to facilitate users to listen to the content if they are unable to read the materials on the go." (Mak et al., 2018)
Rewards	25% (23)	"The app includes gamification entailing an ad hoc character named TIVA—from the Spanish word “posiTIVA,” meaning “positive.” This character grows up and changes every time the caregiver performs the daily activity." (Ferré-Grau et al., 2021)
Simulation	11% (10)	"In the “Lesson” phase, the core skills of the day are introduced via case examples in which the main character experiences the most commonly reported depressive and/or anxiety symptoms." (Ham et al., 2019)
Praise	10% (9)	"Following the completion of six levels pertaining to two beliefs, users may see an encouraging statement such as “Excellent! Now you’ve learned how to better deal with your thoughts and to better recognize the way you overestimate threat.” (Cerea et al., 2021)
Normative Influence	10% (9)	"A peer-support community that is not moderated by professionals, but is rather a place where users can post their thoughts, challenges, and questions and receive support from others using the app." (Moberg et al., 2019)
Social Learning	9% (8)	"After challenge completion, users may choose to anonymously share a short written reflection about the challenge with another user. The app randomly selects another user and sends the written reflection." (Boettcher et al., 2018)
Social Role	7% (6)	"Each skill is taught in a tutorial format by an avatar named “Dr. Joy”, who introduces the player to the game, and explains the processes and skills involved." (Litvin et al., 2020)
Third-Party Endorsements	7% (6)	"Calm is an award-winning mindfulness app that’s available for iOS, Android, and desktop. Developed in 2012, Calm has over 100 million downloads and more than 700,000 5-star reviews." (Laird et al., 2022)
Social Facilitation	5% (5)	"Individuals are also able to choose positive messages (“Message in a Bottle”) to send out to other (anonymous) users." (McCloud et al., 2020)
Authority	4% (4)	"The mobile program complied with the United States Health Insurance Portability and Accountability Act (HIPAA)." (Newman et al., 2021)
Tailoring	3% (3)	Users input their current emotional distress and MoodMission provides a tailored list of five CBT-based activities, called “Missions”." (Bakker et al., 2018)
Cooperation	3% (3)	"Mothers can also reply and answer each other’s questions." (Sawyer et al., 2019)
Surface-Credibility	2% (2)	"Sleep stories narrated by celebrities like Matthew McConaughey and LeBron James." (Huberty et al., 2019)
Social Comparison	2% (2)	"Immediately after each response, a bar graph of the response rate of others is displayed, allowing the participant to see the answers most people chose (i.e., the correct answers)." (Hur et al., 2018)

Table 5

*Prevalence and Application of Novel Persuasive Design Principles in Digital Mental Health Apps*
Novel Persuasive Design Principle	Studies % (N)	Definition	Example of use in app
Staged Disclosure	25% (23)	Presenting information gradually and in a controlled sequence rather than all at once allows users to better absorb and comprehend complex details by breaking them into manageable segments, thereby minimising cognitive overload. Staged disclosure can improve user engagement and retention by maintaining a clear and less overwhelming interaction. This method encourages ongoing use through a gamified approach of 'unlocking levels' and supports deeper understanding over time.	“During the course of the program, participants were prevented from changing the sequence order of the program and were unable to proceed to the next session without completing the previous session.” (Ham et al., 2019)
Goal Setting	15% (13)	Encourages users to establish specific, measurable, and achievable objectives. This approach motivates users by providing clear targets to strive for, which can enhance their sense of purpose, commitment, and progress tracking.	“They then choose from a list of 29 skills any number they want to work on, define their current level of ability and set a goal for how well they want to be able to master a skill." (Boettcher et al., 2018)
Self-Paced	9% (8)	Users proceed through content or tasks at their own pace and according to their preferences. This approach respects users' autonomy and control over their interaction with the system, promoting a more personalised and empowering experience.	"The app was designed to be self-guided. Participants were instructed to go at a pace that suited them." (Linardon et al., 2022)
Limited Use	7% (6)	Limiting the availability or access to certain features, content, or functions within an application or system. This approach is employed to create a sense of exclusivity, thereby increasing the perceived value and desirability of those elements.	"Following the completion of three levels each day, a screen prompting users to stop using the app for that day appears." (Cerea et al., 2021)

Meta-Regression: Persuasive Design Principles and Intervention Efficacy

To explore the relationship between overall intervention efficacy and persuasive design principles, we regressed the number of persuasive design principles used across studies onto effect size estimates (Hedges g) using meta-regression. Figure 3 displays the scatterplot relationship between these variables. Overall, the analysis found no relationship between the number of persuasive design principles used and intervention efficacy across studies, b = .01, SE = .02, 95% CI = [-.04, .05], p = .804. Further, we found no relationship between any persuasive design domain and effectiveness ( Primary task: estimate = − .08, 95% CI [-.17, .01], p = .078), Dialogue: estimate = .06, 95% CI [-.03, .14], p = .184), System Cred: estimate = .001, 95% CI [-.11, .11], p = .981), Social Support: estimate = .05, 95% CI [-.06, .16], p = .375)).

Persuasive Design Principles and Engagement

Seventy-six percent (n = 90) of the studies included in the systematic review provided data on engagement. Among the papers reporting engagement data, a total of 25 distinct engagement indicators were identified. These indicators were grouped into eight overall engagement metrics: 1) Rate of uptake, 2) Time (minutes/hours) spent on app, 3) Days of active use, 4) Logins, 5) Modules Completed, 6) Study Metrics, 7) Messages sent and received, 8) Posts and comments made, 9) Participant Self-Report, and 10) Miscellaneous; see Table 6 for details. The most commonly reported user engagement indicators were: the percentage of participants who completed the entire program or completed the program per protocol, ranging from 8% (Sun et al., 2021) to 100% (Cerea et al., 2021); the average percentage of modules completed, ranging from 30.95% (Newman et al., 2021) to 98% (Torok et al., 2022); and the mean number of logins or visits to the app during the intervention period, ranging from 11.52 logins (Deady et al., 2022) to 106.84 logins (Alvarez et al., 2021).

Table 6

*Range of User Engagement Metrics Across Digital Mental Health Interventions*
User engagement metric	Definition	Number of Studies	Range (lowest and highest reported measurement in this category)
Rate of Uptake	The number of participants that downloaded the intervention/created an account/commenced the intervention/activated the app/ used the app/logged on at least once	11	67.5% (n = 27/40) in Lukas et al., (2021) to 99% (n = 127/128) in Carl et al., (2020)
Time (minutes/hours) spent on app	Mean time spent on app over trial (minutes)	6	41.14 (Flett et al., 2020)- 530.4 minutes (Stolz et al., 2018)
	Median time spent on app over trial (minutes)	2	26.8 (Steare et al., 2020) − 318 minutes (Stolz et al., 2018)
	Mean daily use (minutes)	2	10.8 (Mantani et al., 2018) − 15.25 minutes (Catuara-Solarz et al., 2022)
	Median daily use (minutes)	1	13.93 minutes (Catuara-Solarz et al., 2022)
	Mean hours spent on app per week (hours)	1	3.10 (Huberty et al., 2019)
	Mean weekly use (minutes)	2	37.9 (Huberty et al., 2019) − 103 minutes (Laird et al., 2022)
Days of active use	Mean number of days of active use overall	5	6.38 (Lukas et al., 2021) − 23.07 days (Flett et al., 2020)
Days of active use	Median number of days of active use overall	1	17 days (Catuara-Solarz et al., 2022)
Logins	% of users logged on at least once a day	3	94% (Ebert et al., 2018) – 96% (Heim et al., 2021)
	Average % of logins daily	1	75% (Bucci et al., 2018)
	Mean number of logins/visits over the intervention period	8	11.52 (Deady et al., 2022) − 106.84 logins (Alvarez et al., 2021)
	Median logins to app overall	3	9.5 (Roepke et al., 2015) − 19 logins (Moberg et al., 2019)
Modules Completed	% that completed the first/introductory module	6	60.1% (Carl et al., 2020) – 95% (Carl et al., 2020 ; Ebert et al., 2021)
	% that completed the whole program OR completed the program per protocol	17	7/84 (8%) (Sun et al. 2021) − 25/25 (100%) (Cerea et al. (2021).
	Mean percentage of modules completed	13	12.38/40 (30.95%) (Newman et al., 2021) − 6.84/7 (98%) (Torok et al., 2022)
	Median completion of modules	1	5 modules (5/8 = 63%) (Stolz et al., 2018)
Study Metrics	Number of trial surveys completed	2	48% (vanAubel et al 2020) − 68.7% (Depp et al., 2019)
Study Metrics	Number of participants still engaged at completion of study	2	8.1% at 18months (Alvarez et al., 2021) − 26% at 12 weeks (Ben-Zeev et al., 2018)
Messages sent and received	Mean number of messages sent from user to coach/therapist/clinician	3	10.35 (Newman et al., 2021) − 22 (Raevuori et al., 2021)
Messages sent and received	Mean number of messages sent from coach/therapist/clinician to user	3	25.08 (Newman et al., 2021) − 54 (Raevuori et al., 2021)
Posts and comments made	Mean Number of newsfeed posts and/or comments	1	21.49 (Alvarez et al., 2021)
Posts and comments made	Median Number of newsfeed posts and/or comments	1	7 (Alvarez et al., 2021)
Participant Self-Report	User self-reported engagement	15	N/A
Miscellaneous	Non comparable engagement metrics	11	N/A

The intended meta-regression to explore the relationship between PSD principles and engagement could not be conducted due to the substantial variation in how engagement was conceptualised, measured and reported. Instead, a two-tailed Pearson’s correlation analysis was performed to examine the relationship between number of persuasive design principles used and app engagement across studies. As there was substantial variation in measurements of engagement across studies, this correlation analysis focused on the highest reported engagement metric; percentage of whole program or per-protocol completion (n = 17). No significant relationship was found between number of persuasive design principles used and percentage program completion as an engagement metric across the studies, r(17) = .21, p = .43.

This study found that mental health apps significantly improved clinical outcomes compared to control groups, with a medium effect size (N = 16,728, g = 0.43). Positive effects were observed for depression, anxiety, stress, and body image/eating disorders. However, no significant effects were found for psychosis, suicide/self-harm, postnatal depression, or overall mental health. These results could reflect the greater volume of research for depression, anxiety, stress, and eating disorder apps, highlighting a need for further exploration in under-researched conditions. Importantly, no significant differences in outcomes were identified based on sample type (clinical, subthreshold, or non-clinical), presence of human support, or the clinical approach used (e.g., CBT, mindfulness, psychoeducation).

Across the apps, 79% of persuasive design principles were present, with 100% of apps in the review having at least one principle. Apps most commonly incorporated five principles, with those from the primary task support domain the most frequently used. No studies explicitly stated the use of PSD principles in the development of their app. Meta-regression analysis found no association between the number or domain of persuasive design principles (e.g., primary task, dialogue) and app efficacy. App engagement data were inconsistently reported across studies, preventing a meta-analysis on the relationship between engagement and persuasive design, however, a t-test analysis of the most frequently reported engagement metric—percentage of intervention completers—was conducted and a significant relationship with persuasive design principles was not found.

Although there was broad use of PSD principles across the apps in the examined studies, interestingly none explicitly detailed their use in the app design. Instead, these principles were identified retrospectively through deductive coding of authors app descriptions. This lack of explicit integration may have contributed to the inconclusive findings on the relationship between persuasive design and app efficacy. This lack of explicit nomination of frameworks in app design and development highlights a broader issue; frameworks for describing persuasive principles and features in digital interventions are often absent or inconsistently detailed. As a result, these features may be present but unreported, or they may vary significantly in the quality of execution. To enable accurate assessment of both their presence and quality, there is a need for a standardised framework and consensus on how to describe engagement-enhancing features in digital mental health apps.

To systematically address the challenge of engagement in digital mental health, more intentional incorporation of frameworks such as persuasive design principles and documentation of such frameworks during app development is urgently needed.

Our findings contrast with earlier research, which has reported both positive (Kelders et al., 2021; McCall et al., 2021; Orji & Moffatt., 2018; Wu et al., 2021) and negative (Wu et al., 2021) associations between PSD and engagement or efficacy outcomes. These discrepancies likely arise from variations in the digital intervention studied and the coding of design principles. For example, Kelders et al. (2012) and Wu et al. (2021) relied on descriptions of interventions that were explicitly reported in included publications, while we consulted supplementary materials, such as protocols, development papers, and websites, potentially allowing for a more comprehensive assessment of the persuasive design principles and features utilised in each intervention. However, due to the inconsistency of reporting, it is challenging to accurately determine how and to what extent each study incorporated persuasive system design (PSD) principles without explicit disclosure from the authors. Additionally, different technology modalities (e.g., web-based interventions vs. apps) may influence how users interact with persuasive principles, leading to different relationships between design features and outcomes. Kelder et al. (2012) focused on 83 web-based interventions, as opposed to apps, that focused on eHealth more broadly (chronic conditions, smoking cessation etc). Wu and colleagues (2021) and McHall and colleagues (2021) both focused solely on depression and anxiety. Finally, differences in study design, measurement methods, or analytical approaches may also contribute to these discrepancies. Our study, using a robust methodology with RCT-tested apps across all mental health conditions and a total sample size of 16,728, offered greater statistical power and subgroup analysis, enhancing the reliability and precision of our effect estimates.

The limited evidence supporting PSD principles to enhance engagement in the current study may be due to several factors. As mentioned, inadequate reporting of how persuasive design principles are incorporated into apps may have led to missed or misattribution of PSD principles to apps. This could also be a threshold issue, where a principle is minimally applied and marked as present, while another app may implement the same principle extensively and effectively, yet both receive the same score. Second, substantial variability in engagement measurement across studies limited our ability to comprehensively explore the relationship between PSD principles and engagement. Third, the relationship between the number of PSD principles and engagement or efficacy may not be linear, making it more difficult to identify. For example, certain combinations of PSD principles may be more effective in fostering engagement and efficacy for specific subgroups of users. Fourth, other variables influencing engagement (e.g., overall app aesthetics and design, human support, user characteristics, social reciprocity, social ranking, variable unpredictable rewards, etc) may be masking the relationship between the PSD principles researched in this study and engagement. Finally, it is possible that PSD principles do not directly impact engagement or efficacy. However, this seems unlikely given the well-established role of user experience and design in shaping how users engage with technology. For example, studies indicate that user-centred design improves both usability and engagement by addressing key factors such as visual hierarchy, layout, and interaction patterns (Huang & Benyoucef, 2023).

This considerable variability around engagement metrics concerned how engagement was both defined and reported across studies. Notably, 24% of the included studies did not report engagement metrics at all, a considerable proportion given the importance of the engagement challenge in digital interventions. This aligns with findings by Lipschitz et al. (2022), who highlighted the low rates of engagement reporting in digital mental health interventions for depression, where only 64% of studies reported daily usage and just 23% provided retention rates for the final treatment week. The present study extends these concerns to mental health apps more broadly, indicating that inconsistent and selective reporting of engagement metrics may skew perceived levels of user interaction. Such selective reporting can artificially inflate engagement data, potentially overestimating the use, relevance, and even efficacy of interventions. These discrepancies underscore the need for standardized engagement metrics to ensure accurate, comprehensive reporting and a clearer understanding of user interaction with digital mental health interventions.

The focus on engagement metrics with digital interventions rests on the understanding that engagement is necessary for the intervention to be effective. Engagement metrics are, therefore, crucial for ensuring that users are exposed to the intervention content and essential for attributing positive outcomes to the intervention in question (Lipschitz et al 2022). However, to get a true understanding of the relationship between engagement and effectiveness, we must also understand the behaviour change mechanisms of the intervention, that is, how the intervention drives real-world change. The factors influencing engagement with the digital intervention, such as persuasive design, do not fully align with those driving real-world behaviour change, which are often rooted in deeper motivations, self-efficacy, and the perceived relevance and feasibility of recommended behaviours (Michie et al., 2011). Despite the aim of promoting behaviour change, Orji and Moffat (2016) found that 55% of studies did not link mental health interventions to any behaviour change theory. This absence of theoretical underpinning was also observed in the present study, where the extraction of behaviour change theories underlying the reviewed studies was halted due to insufficient reporting.

Therefore, to advance the field, we propose several recommendations aimed at addressing gaps in the development, reporting, and evaluation of digital mental health interventions. Establishing standardised engagement metrics is crucial for enabling reliable comparisons and facilitating meta-analyses. We recommend adopting guidelines like CONSORT-EHEALTH (Eysenbach & CONSORT-EHEALTH Group, 2012) to improve data consistency and provide clearer insights into how persuasive design principles could influence engagement and outcomes. We endorse and propose to extend the five-point checklist from Lipschitz et al. (2022), which includes metrics such as adherence, rate of uptake, level-of-use, duration-of-use, and number of completers, by incorporating two additional measures aimed at providing a more comprehensive understanding of intervention impact. First, the description of interventions should include detailed information on the design principles and features of the platform that promote engagement; authors should explicitly state the frameworks (e.g., PSD) employed. Second, authors should explicitly link the intervention to an overarching model of behaviour change. By systematically reporting these principles alongside consistent engagement metrics, researchers can better establish links between design elements, user engagement outcomes, real-world behaviour change, theoretical models, and clinical outcomes. This approach will facilitate deeper understanding of user engagement and what drives therapeutic change.

Limitations

This study had several notable limitations. Firstly, the use of persuasive principles was not explicitly reported in any studies included in the analysis. Although we utilised a well-defined framework and an expert team of user experience designers for coding, many reviewed apps were not commercially available, limiting their comprehensive description and accessibility for analysis. As a result, most apps were coded retrospectively based on available literature. This method raises concerns that some apps may have employed persuasive principles not described in their app descriptions, potentially affecting assessments of effectiveness and outcomes. Additionally, there is uncertainty about the extent to which persuasive principles were actually incorporated, even among those apps that were coded for it. This limitation highlights the urgent need for greater transparency in app reporting, especially for apps that are not commercially available, to facilitate more detailed evaluations of how persuasive design principles impact mental health app outcomes.

Additionally, we could only conduct a single correlation analysis between PSD and one engagement measure. With only 76% of studies reporting on engagement or adherence, and those metrics being highly heterogeneous, a meta-analysis was not feasible. This inconsistency in reporting and operationalizing engagement metrics significantly hinders progress in understanding app and digital intervention engagement. Consequently, these limitations constrained our ability to comprehensively evaluate the relationship between persuasive design principles and app engagement and effectiveness. Despite these limitations, to our knowledge this is the most extensive examination to date on the relationship between PSD principles, engagement and efficacy in digital mental health. We included 119 RCTs, coded all persuasive design principles with high inter-rater reliability, reviewed all protocols and associated reports, including downloading the apps when available, and categorised all engagement data reported into meaningful categories.

This study provides the most extensive and systematic evaluation to date of the relationship between persuasive design principles, engagement, and efficacy in digital mental health apps. Results identified that mental health apps are moderately effective for depression, anxiety, stress, and body image/eating disorders but show limited efficacy for psychosis, suicide/self-harm, and postnatal depression, highlighting the need for further research in these areas.

While persuasive design principles are intended to enhance user engagement, our study found limited evidence of their direct impact, likely due to inconsistent reporting, variability in their application, and a lack of standardised engagement metrics. Further, due to the rapid development of technology and its capabilities, persuasive principle frameworks require revision and updating to include strategies such as staged disclosure as identified in this study. These findings underscore critical limitations in the field that require immediate attention. To address the historical issue of poor engagement with digital mental health interventions, we need a concerted effort to harmonise reporting of engagement, and the descriptions of persuasive strategies and behaviour change frameworks used to underpin it. Addressing these methodological gaps is essential to enabling large-scale data pooling, which will facilitate a more nuanced understanding of the factors influencing engagement, tailored to specific populations and contexts. Without such improvements to lift the field, the key challenge of engagement in digital mental health will remain intractable, limiting real-world impact.

Competing Interests

No competing interests to declare

Author Contribution

L.V., J.D.X.H., J.N and M.A.J. conceptualised and designed the study. LV., J.D.X.H., J.N and M.A.J. developed the search strategy. L.V and J.D.X.H conducted title/abstract and full text screening. L.V., S.O., R.P.S., M.S.V., and W.P conducted data extraction. LV., KB and LB conducted persuasive design principle coding. J.D.X.H., S.O., and V.L conducted the analyses. T.W., C.M., S.M., I.B and L.V conducted the risk of bias assessment. L.V wrote the first draft of the manuscript, with input from J.N., S.B., J.D.X.H., M.A.J and S.C. All authors revised the manuscript and approved the final version of the submitted manuscript.

Acknowledgments

No funding to declare

Data Availability

The datasets analysed during for this review are available from the corresponding author on request.

Akin-Sari, B., Inozu, M., Haciomeroglu, A. B., Cekci, B. C., Uzumcu, E., & Doron, G. (2022). Cognitive training via a mobile application to reduce obsessive-compulsive-related distress and cognitions during the COVID-19 outbreaks: a randomized controlled trial using a subclinical cohort. Behavior Therapy, 53(5), 776-792.
Al-Refae, M., Al-Refae, A., Munroe, M., Sardella, N. A., & Ferrari, M. (2021). A self-compassion and mindfulness-based cognitive mobile intervention (Serene) for depression, anxiety, and stress: Promoting adaptive emotional regulation and wisdom. Frontiers in Psychology, 12, 648087.
Alvarez-Jimenez, M., et al. (2021). "The Horyzons project: a randomized controlled trial of a novel online social therapy to maintain treatment effects from specialist first-episode psychosis services." World Psychiatry 20(2): 233-243.
Alvarez‐Jimenez, M., Nicholas, J., Valentine, L., Liu, P., Mangelsdorf, S., Baker, S., ... & Cross, S. P. (2024). A national evaluation of a multi‐modal, blended, digital intervention integrated within Australian youth mental health services. Acta Psychiatrica Scandinavica.
Andersson, C., Bergsten, K. L., Lilliengren, P., Norbäck, K., Rask, K., Einhorn, S., & Osika, W. (2021). The effectiveness of smartphone compassion training on stress among Swedish university students: A pilot randomized trial. Journal of Clinical Psychology, 77(4), 927-945.
Araya, R., Menezes, P. R., Claro, H. G., Brandt, L. R., Daley, K. L., Quayle, J., ... & Miranda, J. J. (2021). Effect of a digital intervention on depressive symptoms in patients with comorbid hypertension or diabetes in Brazil and Peru: two randomized clinical trials. Jama, 325(18), 1852-1862.
Arean, P. A., Hallgren, K. A., Jordan, J. T., Gazzaley, A., Atkins, D. C., Heagerty, P. J., & Anguera, J. A. (2016). The use and effectiveness of mobile apps for depression: results from a fully remote clinical trial. Journal of medical Internet research, 18(12), e330.
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2018). A randomized controlled trial of three smartphone apps for enhancing public mental health. Behaviour Research and Therapy, 109, 75-83.
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019). Objective user engagement with mental health apps: systematic search and panel-based usage analysis. Journal of medical Internet research, 21(9), e14567.
Baumel, A. (2022). Therapeutic activities as a link between program usage and clinical outcomes in digital mental health interventions: a proposed research framework. Journal of Technology in Behavioral Science, 7(2), 234-239.
Bell, I., Ellinghaus, C., Pot-Kolder, R., Nicholas, J., Tennant, M., Macmillan, I. C., ... & Thompson, A. D. (2023). Attitudes towards the use of virtual reality within early psychosis services amongst clinicians and young people. Early Intervention in Psychiatry, 1(S1), 141.
Ben-Zeev, D., Brian, R. M., Jonathan, G., Razzano, L., Pashka, N., Carpenter-Song, E., ... & Scherer, E. A. (2018). Mobile health (mHealth) versus clinic-based group intervention for people with serious mental illness: a randomized controlled trial. Psychiatric Services, 69(9), 978-985.
Ben-Zeev, D., Chander, A., Tauscher, J., Buck, B., Nepal, S., Campbell, A., & Doron, G. (2021). A smartphone intervention for people with serious mental illness: fully remote randomized controlled trial of CORE. Journal of medical Internet research, 23(11), e29201.
Birney, A. J., Gunn, R., Russell, J. K., & Ary, D. V. (2016). MoodHacker mobile web app with email for adults to self-manage mild-to-moderate depression: randomized controlled trial. JMIR mHealth and uHealth, 4(1), e4231.
Boettcher, J., Magnusson, K., Marklund, A., Berglund, E., Blomdahl, R., Braun, U., ... & Carlbring, P. (2018). Adding a smartphone app to internet-based self-help for social anxiety: A randomized controlled trial. Computers in Human Behavior, 87, 98-108.
Bond, R. R., Mulvenna, M. D., Potts, C., O’Neill, S., Ennis, E., & Torous, J. (2023). Digital transformation of mental health services. Npj Mental Health Research, 2(1), 13.
Borghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., ... & Sorkin, D. H. (2021). Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. Journal of medical Internet research, 23(3), e24387.
Børøsund, E., Ehlers, S. L., Varsi, C., Clark, M. M., Andrykowski, M. A., Cvancarova, M., & Solberg Nes, L. (2020). Results from a randomized controlled trial testing StressProffen; an application‐based stress‐management intervention for cancer survivors. Cancer medicine, 9(11), 3775-3785.
Bruehlman-Senecal, E., Hook, C. J., Pfeifer, J. H., FitzGerald, C., Davis, B., Delucchi, K. L., ... & Ramo, D. E. (2020). Smartphone app to address loneliness among college students: pilot randomized controlled trial. JMIR Mental Health, 7(10), e21496.
Bruhns, A., Lüdtke, T., Moritz, S., & Bücker, L. (2021). A mobile-based intervention to increase self-esteem in students with depressive symptoms: randomized controlled trial. JMIR mHealth and uHealth, 9(7), e26498.
Bucci, S., Barrowclough, C., Ainsworth, J., Machin, M., Morris, R., Berry, K., ... & Haddock, G. (2018). Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophrenia bulletin, 44(5), 1070-1080.
Bush, N. E., Smolenski, D. J., Denneson, L. M., Williams, H. B., Thomas, E. K., & Dobscha, S. K. (2017). A virtual hope box: randomized controlled trial of a smartphone app for emotional regulation and coping with distress. Psychiatric Services, 68(4), 330-336.
Carl, J. R., Miller, C. B., Henry, A. L., Davis, M. L., Stott, R., Smits, J. A., ... & Espie, C. A. (2020). Efficacy of digital cognitive behavioral therapy for moderate‐to‐severe symptoms of generalized anxiety disorder: A randomized controlled trial. Depression and anxiety, 37(12), 1168-1178.
Catuara-Solarz, S., Skorulski, B., Estella-Aguerri, I., Avella-Garcia, C. B., Shepherd, S., Stott, E., ... & Dix, S. (2022). The efficacy of “Foundations,” a digital mental health app to improve mental well-being during COVID-19: proof-of-principle randomized controlled trial. JMIR mHealth and uHealth, 10(7), e30976.
Cerea, S., Ghisi, M., Bottesi, G., Carraro, E., Broggio, D., & Doron, G. (2020). Reaching reliable change using short, daily, cognitive training exercises delivered on a mobile application: The case of Relationship Obsessive Compulsive Disorder (ROCD) symptoms and cognitions in a subclinical cohort. Journal of Affective Disorders, 276, 775-787.
Cerea, S., Ghisi, M., Bottesi, G., Manoli, T., Carraro, E., & Doron, G. (2021). Cognitive behavioral training using a mobile application reduces body image-related symptoms in high-risk female university students: a randomized controlled study. Behavior Therapy, 52(1), 170-182.
Cerea, S., Doron, G., Manoli, T., Patania, F., Bottesi, G., & Ghisi, M. (2022). Cognitive training via a mobile application to reduce some forms of body dissatisfaction in young females at high-risk for body image disorders: a randomized controlled trial. Body image, 42, 297-306.
Chu, A., Rose, T. M., Gundrum, D. A., McMorris, T. E., Klausner, E. A., Lang, L. A., & Shan, G. (2022). Evaluating the effects of a mindfulness mobile application on student pharmacists’ stress, burnout, and mindfulness. American Journal of Health-System Pharmacy, 79(8), 656-664.
Cochran, W. G. (1954). The combination of estimates from different experiments. Biometrics, 10(1), 101-129. https://doi.org/10.2307/3001666
Chan, K. L., Leung, W. C., Tiwari, A., Or, K. L., & Ip, P. (2019). Using smartphone-based psychoeducation to reduce postnatal depression among first-time mothers: randomized controlled trial. JMIR mHealth and uHealth, 7(5), e12794.
Chan, C. S., Wong, C. Y., Branda, Y. M., Hui, V. K., Ho, F. Y., & Cuijpers, P. (2023). Treating depression with a smartphone-delivered self-help cognitive behavioral therapy for insomnia: a parallel-group randomized controlled trial. Psychological medicine, 53(5), 1799-1813.
Christoforou, M., Sáez Fonseca, J. A., & Tsakanikos, E. (2017). Two novel cognitive behavioral therapy–based mobile apps for agoraphobia: randomized controlled trial. Journal of medical Internet research, 19(11), e398.
Cole-Lewis, H., Ezeanochie, N., & Turgiss, J. (2019). Understanding health behavior technology engagement: pathway to measuring digital behavior change interventions. JMIR formative research, 3(4), e14052.
Corscadden, L., Callander, E. J., & Topp, S. M. (2019). Who experiences unmet need for mental health services and what other barriers to accessing health care do they face? F indings from Australia and Canada. The International Journal of Health Planning and Management, 34(2), 761-772.
Cross, S., Piper, S. E., Davenport, T. A., Milton, A., Iorfino, F., Rici, C., ... & Hickie, I. (2019). Implementation study of a prototypic e-clinic being integrated into youth mental health services: staff experiences and reported service quality improvements. Med J Aust, 211, S30-6.
Cross, S., Nicholas, J., Mangelsdorf, S., Valentine, L., Baker, S., McGorry, P., ... & Alvarez-Jimenez, M. (2023). Developing a Theory of Change for a Digital Youth Mental Health Service (Moderated Online Social Therapy): Mixed Methods Knowledge Synthesis Study. JMIR Formative Research, 7, e49846.
Cuijpers, P., Heim, E., Abi Ramia, J., Burchert, S., Carswell, K., Cornelisz, I., ... & El Chammay, R. (2022). Effects of a WHO-guided digital health intervention for depression in Syrian refugees in Lebanon: A randomized controlled trial. PLoS medicine, 19(6), e1004025.
Cuijpers, P., Heim, E., Abi Ramia, J., Burchert, S., Carswell, K., Cornelisz, I., ... & El Chammay, R. (2022). Guided digital health intervention for depression in Lebanon: randomised trial. BMJ Ment Health, 25(e1), e34-e40.
Dagöö, J., Asplund, R. P., Bsenko, H. A., Hjerling, S., Holmberg, A., Westh, S., ... & Andersson, G. (2014). Cognitive behavior therapy versus interpersonal psychotherapy for social anxiety disorder delivered via smartphone and computer: a randomized controlled trial. Journal of anxiety disorders, 28(4), 410-417.
Dahne, J., Collado, A., Lejuez, C. W., Risco, C. M., Diaz, V. A., Coles, L., ... & Carpenter, M. J. (2019). Pilot randomized controlled trial of a Spanish-language Behavioral Activation mobile app (¡ Aptívate!) for the treatment of depressive symptoms among united states Latinx adults with limited English proficiency. Journal of affective disorders, 250, 210-217.
Deady, M., Glozier, N., Calvo, R., Johnston, D., Mackinnon, A., Milne, D., ... & Harvey, S. B. (2022). Preventing depression using a smartphone app: a randomized controlled trial. Psychological medicine, 52(3), 457-466.
de Beurs, D., van Bruinessen, I., Noordman, J., Friele, R., & van Dulmen, S. (2017). Active involvement of end users when developing web-based mental health interventions. Frontiers in psychiatry, 8, 72.
De Kock, J. H., Latham, H. A., Cowden, R. G., Cullen, B., Narzisi, K., Jerdan, S., ... & Eze, J. (2022). Brief digital interventions to support the psychological well-being of NHS staff during the COVID-19 pandemic: 3-arm pilot randomized controlled trial. JMIR mental health, 9(4), e34002.
Depp, C. A., Ceglowski, J., Wang, V. C., Yaghouti, F., Mausbach, B. T., Thompson, W. K., & Granholm, E. L. (2015). Augmenting psychoeducation with a mobile intervention for bipolar disorder: a randomized controlled trial. Journal of affective disorders, 174, 23-30.
Depp, C. A., Perivoliotis, D., Holden, J., Dorr, J., & Granholm, E. L. (2019). Single-session mobile-augmented intervention in serious mental illness: a three-arm randomized controlled trial. Schizophrenia Bulletin, 45(4), 752-762.
Di Simplicio, M., Appiah‐Kusi, E., Wilkinson, P., Watson, P., Meiser‐Stedman, C., Kavanagh, D. J., & Holmes, E. A. (2020). Imaginator: A proof‐of‐concept feasibility trial of a brief imagery‐based psychological intervention for young people who self‐Harm. Suicide and Life‐Threatening Behavior, 50(3), 724-740.
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x
Ebenfeld, L., Lehr, D., Ebert, D. D., Kleine Stegemann, S., Riper, H., Funk, B., & Berking, M. (2021). Evaluating a hybrid web-based training program for panic disorder and agoraphobia: randomized controlled trial. Journal of medical Internet research, 23(3), e20829.
Ebert, D. D., Buntrock, C., Lehr, D., Smit, F., Riper, H., Baumeister, H., ... & Berking, M. (2018). Effectiveness of web-and mobile-based treatment of subthreshold depression with adherence-focused guidance: a single-blind randomized controlled trial. Behavior Therapy, 49(1), 71-83.
Ebert, D. D., Franke, M., Zarski, A. C., Berking, M., Riper, H., Cuijpers, P., ... & Lehr, D. (2021). Effectiveness and moderators of an internet-based mobile-supported stress management intervention as a universal prevention approach: randomized controlled trial. Journal of medical Internet research, 23(12), e22107.
Economides, M., Martman, J., Bell, M. J., & Sanderson, B. (2018). Improvements in stress, affect, and irritability following brief use of a mindfulness-based smartphone app: a randomized controlled trial. Mindfulness, 9(5), 1584-1593.
Eichstedt, J. A., Turcotte, K., Golden, G., Arbuthnott, A. E., Chen, S., Collins, K., ... & Reid, G. J. (2024). Waitlist management in child and adolescent mental health care: A scoping review. Children and Youth Services Review, 107529.
Engel, L., et al. (2023). "The Cost-Effectiveness of a Novel Online Social Therapy to Maintain Treatment Effects From First-Episode Psychosis Services: Results From the Horyzons Randomized Controlled Trial." Schizophrenia Bulletin: sbad071.
Enock, P. M., Hofmann, S. G., & McNally, R. J. (2014). Attention bias modification training via smartphone to reduce social anxiety: A randomized, controlled multi-session experiment. Cognitive therapy and research, 38, 200-216.
Erbe, D., Eichert, H. C., Riper, H., & Ebert, D. D. (2017). Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: systematic review. Journal of medical Internet research, 19(9), e306.
Everitt, N., Broadbent, J., Richardson, B., Smyth, J. M., Heron, K., Teague, S., & Fuller-Tyszkiewicz, M. (2021). Exploring the features of an app-based just-in-time intervention for depression. Journal of affective disorders, 291, 279-287.
Eysenbach, G., & Consort-EHEALTH Group. (2011). CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. Journal of medical Internet research, 13(4), e1923.
Ferré-Grau, C., Raigal-Aran, L., Lorca-Cabrera, J., Lluch-Canut, T., Ferré-Bergadà, M., Lleixá-Fortuño, M., ... & Albacar-Riobóo, N. (2021). A mobile app–based intervention program for nonprofessional caregivers to promote positive mental health: Randomized controlled trial. JMIR mHealth and uHealth, 9(1), e21708.
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018). Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. Journal of medical Internet research, 20(6), e199.
Flett, J. A., Hayne, H., Riordan, B. C., Thompson, L. M., & Conner, T. S. (2019). Mobile mindfulness meditation: a randomised controlled trial of the effect of two popular apps on mental health. Mindfulness, 10, 863-876.
Flett, J. A., Conner, T. S., Riordan, B. C., Patterson, T., & Hayne, H. (2020). App-based mindfulness meditation for psychological distress and adjustment to college in incoming university students: a pragmatic, randomised, waitlist-controlled trial. Psychology & health, 35(9), 1049-1074.
Fogg, B. J., Cueller, G., & Danielson, D. (2007). Motivating, influencing, and persuading users: An introduction to captology. In The human-computer interaction handbook (pp. 159-172). CRC press.
Fogg, B. J. (1998, January). Persuasive computers: perspectives and research directions. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 225-232).
Franklin, J. C., Fox, K. R., Franklin, C. R., Kleiman, E. M., Ribeiro, J. D., Jaroszewski, A. C., ... & Nock, M. K. (2016). A brief mobile app reduces nonsuicidal and suicidal self-injury: Evidence from three randomized controlled trials. Journal of Consulting and Clinical Psychology, 84(6), 544.
Fuller-Tyszkiewicz, M., Richardson, B., Little, K., Teague, S., Hartley-Clark, L., Capic, T., ... & Hutchinson, D. (2020). Efficacy of a smartphone app intervention for reducing caregiver stress: randomized controlled trial. JMIR mental health, 7(7), e17541.
Gao, M., Roy, A., Deluty, A., Sharkey, K. M., Hoge, E. A., Liu, T., & Brewer, J. A. (2022). Targeting anxiety to improve sleep disturbance: a randomized clinical trial of app-based mindfulness training. Psychosomatic medicine, 84(5), 632-642.
Gan, D. Z., McGillivray, L., Han, J., Christensen, H., & Torok, M. (2021). Effect of engagement with digital interventions on mental health outcomes: a systematic review and meta-analysis. Frontiers in digital health, 3, 764079.
Gega, L., Jankovic, D., Saramago, P., Marshall, D., Dawson, S., Brabyn, S., ... & Bojke, L. (2022). Digital interventions in mental health: evidence syntheses and economic modelling. Health Technology Assessment (Winchester, England), 26(1), 1.
Gentili, A., Failla, G., Melnyk, A., Puleo, V., Tanna, G. L. D., Ricciardi, W., & Cascini, F. (2022). The cost-effectiveness of digital health interventions: a systematic review of the literature. Frontiers in Public Health, 10, 787135.
Ghanbari, E., Yektatalab, S., & Mehrabi, M. (2021). Effects of psychoeducational interventions using mobile apps and mobile-based online group discussions on anxiety and self-esteem in women with breast cancer: randomized controlled trial. JMIR mHealth and uHealth, 9(5), e19262.
Gnanapragasam, S. N., Tinch-Taylor, R., Scott, H. R., Hegarty, S., Souliou, E., Bhundia, R., ... & Wessely, S. (2023). Multicentre, England-wide randomised controlled trial of the ‘Foundations’ smartphone application in improving mental health and well-being in a healthcare worker population. The British Journal of Psychiatry, 222(2), 58-66.
Graham, A. K., Greene, C. J., Kwasny, M. J., Kaiser, S. M., Lieponis, P., Powell, T., & Mohr, D. C. (2020). Coached mobile app platform for the treatment of depression and anxiety among primary care patients: a randomized clinical trial. JAMA psychiatry, 77(9), 906-914.
Greer, J. A., Jacobs, J., Pensak, N., MacDonald, J. J., Fuh, C. X., Perez, G. K., ... & Temel, J. S. (2019). Randomized trial of a tailored cognitive‐behavioral therapy mobile application for anxiety in patients with incurable cancer. The oncologist, 24(8), 1111-1120.
Ham, K., Chin, S., Suh, Y. J., Rhee, M., Yu, E. S., Lee, H. J., ... & Chung, K. M. (2019). Preliminary results from a randomized controlled study for an app-based cognitive behavioral therapy program for depression and anxiety in cancer patients. Frontiers in psychology, 10, 1592.
Harrer, M., Apolinário-Hagen, J., Fritsche, L., Salewski, C., Zarski, A. C., Lehr, D., ... & Ebert, D. D. (2021). Effect of an internet-and app-based stress intervention compared to online psychoeducation in university students with depressive symptoms: Results of a randomized controlled trial. Internet interventions, 24, 100374.
Heim, E., Abi Ramia, J., Abi Hana, R., Burchert, S., Carswell, K., Cornelisz, I., ... & Van't Hof, E. (2021). Step-by-step: feasibility randomised controlled trial of a mobile-based intervention for depression among populations affected by adversity in Lebanon. Internet interventions, 24, 100380.
Hensler, I., Sveen, J., Cernvall, M., & Arnberg, F. K. (2022). Efficacy, benefits, and harms of a self-management app in a Swedish trauma-exposed community sample (PTSD coach): Randomized controlled trial. Journal of Medical Internet Research, 24(3), e31419.
Hicks, M. D., Braden, L. A., Walsh, E. M., Greene, B. J., & Grayson, J. W. (2022). Mobile meditation for improving quality of life, anxiety and depression among surgical residents and faculty. The Journal of Laryngology & Otology, 136(11), 1034-1038.
Hides, L., Dingle, G., Quinn, C., Stoyanov, S. R., Zelenko, O., Tjondronegoro, D., ... & Kavanagh, D. J. (2019). Efficacy and outcomes of a music-based emotion regulation mobile app in distressed young people: randomized controlled trial. JMIR mHealth and uHealth, 7(1), e11482.
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ 327(7414), 557-560. https://doi.org/10.1136/bmj.327.7414.557
Hildebrandt, T., Michaelides, A., Mackinnon, D., Greif, R., DeBar, L., & Sysko, R. (2017). Randomized controlled trial comparing smartphone assisted versus traditional guided self‐help for adults with binge eating. International Journal of Eating Disorders, 50(11), 1313-1322.
Hirshberg, M. J., Frye, C., Dahl, C. J., Riordan, K. M., Vack, N. J., Sachs, J., ... & Goldberg, S. B. (2022). A randomized controlled trial of a smartphone-based well-being training in public school system employees during the COVID-19 pandemic. Journal of educational psychology, 114(8), 1895.
Huberty, J., Green, J., Glissmann, C., Larkey, L., Puzia, M., & Lee, C. (2019). Efficacy of the mindfulness meditation mobile app “calm” to reduce stress among college students: Randomized controlled trial. JMIR mHealth and uHealth, 7(6), e14273.
Huberty, J., Puzia, M., Vlisides-Henry, R., Larkey, L., Irwin, M., & Vranceanu, A. M. (2021, April). EFFECTS OF A MEDITATION MOBILE APP ON DEPRESSION AND ANXIETY IN ADULTS WITH SLEEP DISTURBANCE: A RANDOMIZED CONTROLLED TRIAL. In ANNALS OF BEHAVIORAL MEDICINE (Vol. 55, pp. S174-S174). JOURNALS DEPT, 2001 EVANS RD, CARY, NC 27513 USA: OXFORD UNIV PRESS INC.
Hur, J. W., Kim, B., Park, D., & Choi, S. W. (2018). A scenario-based cognitive behavioral therapy mobile app to reduce dysfunctional beliefs in individuals with depression: a randomized controlled trial. Telemedicine and e-Health, 24(9), 710-716.
Hwang, W. J., & Jo, H. H. (2019). Evaluation of the effectiveness of mobile app-based stress-management program: a randomized controlled trial. International journal of environmental research and public health, 16(21), 4270.
Hwang, H., Kim, S. M., Netterstrøm, B., & Han, D. H. (2022). The efficacy of a smartphone-based App on stress reduction: randomized controlled trial. Journal of medical Internet research, 24(2), e28703.
Ivanova, E., Lindner, P., Ly, K. H., Dahlin, M., Vernmark, K., Andersson, G., & Carlbring, P. (2016). Guided and unguided Acceptance and Commitment Therapy for social anxiety disorder and/or panic disorder provided via the Internet and a smartphone application: A randomized controlled trial. Journal of anxiety disorders, 44, 27-35.
Jannati, N., Mazhari, S., Ahmadian, L., & Mirzaee, M. (2020). Effectiveness of an app-based cognitive behavioral therapy program for postpartum depression in primary care: A randomized controlled trial. International Journal of Medical Informatics, 141, 104145.
Kageyama, K., Kato, Y., Mesaki, T., Uchida, H., Takahashi, K., Marume, R., ... & Hirao, K. (2021). Effects of video viewing smartphone application intervention involving positive word stimulation in people with subthreshold depression: a pilot randomized controlled trial. Journal of affective disorders, 282, 74-81.
Kahn, J. R., Collinge, W., & Soltysik, R. (2016). Post-9/11 veterans and their partners improve mental health outcomes with a self-directed mobile and web-based wellness training program: a randomized controlled trial. Journal of medical Internet research, 18(9), e255.
Kollei, I., Lukas, C. A., Loeber, S., & Berking, M. (2017). An app-based blended intervention to reduce body dissatisfaction: A randomized controlled pilot study. Journal of Consulting and Clinical Psychology, 85(11), 1104.
Kubo, A., Kurtovich, E., McGinnis, M., Aghaee, S., Altschuler, A., Quesenberry Jr, C., ... & Avins, A. L. (2019). A randomized controlled trial of mHealth mindfulness intervention for cancer patients and informal cancer caregivers: a feasibility study within an integrated health care delivery system. Integrative cancer therapies, 18, 1534735419850634.
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012). Persuasive system design does matter: a systematic review of adherence to web-based interventions. Journal of medical Internet research, 14(6), e152.
Kranenburg, L. W., Gillis, J., Mayer, B., & Hoogendijk, W. J. (2022). The effectiveness of a nonguided mindfulness app on perceived stress in a nonclinical Dutch population: randomized controlled trial. JMIR Mental Health, 9(3), e32123.
Kruzan, K. P., Whitlock, J., Bazarova, N. N., Bhandari, A., & Chapman, J. (2022). Use of a mobile peer support app among young people with nonsuicidal self-injury: small-scale randomized controlled trial. JMIR formative research, 6(1), e26526.
Laird, B., Puzia, M., Larkey, L., Ehlers, D., & Huberty, J. (2022). A mobile app for stress management in middle-aged men and women (Calm): feasibility randomized controlled trial. JMIR formative research, 6(5), e30294.
Lattie, E. G., Adkins, E. C., Winquist, N., Stiles-Shields, C., Wafford, Q. E., & Graham, A. K. (2019). Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: systematic review. Journal of medical Internet research, 21(7), e12869.
Lee, R. A., & Jung, M. E. (2018). Evaluation of an mhealth app (destressify) on university students’ mental health: pilot trial. JMIR mental health, 5(1), e8324.
Levin, M. E., Hicks, E. T., & Krafft, J. (2022). Pilot evaluation of the stop, breathe & think mindfulness app for student clients on a college counseling center waitlist. Journal of American college health, 70(1), 165-173.
Linardon, J., Shatte, A., Rosato, J., & Fuller-Tyszkiewicz, M. (2022). Efficacy of a transdiagnostic cognitive-behavioral intervention for eating disorder psychopathology delivered through a smartphone app: a randomized controlled trial. Psychological Medicine, 52(9), 1679-1690.
Linardon, J., Shatte, A., McClure, Z., & Fuller-Tyszkiewicz, M. (2023). A broad v. focused digital intervention for recurrent binge eating: A randomized controlled non-inferiority trial. Psychological Medicine, 53(10), 4580-4591.
Linardon, J., Torous, J., Firth, J., Cuijpers, P., Messer, M., & Fuller‐Tyszkiewicz, M. (2024). Current evidence on the efficacy of mental health smartphone apps for symptoms of depression and anxiety. A meta‐analysis of 176 randomized controlled trials. World Psychiatry, 23(1), 139-149.
Lipschitz, J. M., Van Boxtel, R., Torous, J., Firth, J., Lebovitz, J. G., Burdick, K. E., & Hogan, T. P. (2022). Digital mental health interventions for depression: scoping review of user engagement. Journal of medical Internet research, 24(10), e39204.
Lipschitz, Jessica M., Chelsea K. Pike, Timothy P. Hogan, Susan A. Murphy, and Katherine E. Burdick. "The engagement problem: A review of engagement with digital mental health interventions and recommendations for a path forward." Current treatment options in psychiatry 10, no. 3 (2023): 119-135.
Litvin, S., Saunders, R., Maier, M. A., & Lüttke, S. (2020). Gamification as an approach to improve resilience and reduce attrition in mobile mental health interventions: A randomized controlled trial. PLoS one, 15(9), e0237220.
Lukas, C. A., & Berking, M. (2021). Blending group-based psychoeducation with a smartphone intervention for the reduction of depressive symptoms: results of a randomized controlled pilot study. Pilot and feasibility studies, 7, 1-8.
Lukas, C. A., Eskofier, B., & Berking, M. (2021). A gamified smartphone-based intervention for depression: randomized controlled pilot trial. JMIR Mental Health, 8(7), e16643.
Ly, K. H., Trüschel, A., Jarl, L., Magnusson, S., Windahl, T., Johansson, R., ... & Andersson, G. (2014). Behavioural activation versus mindfulness-based guided self-help treatment administered through a smartphone application: a randomised controlled trial. BMJ open, 4(1), e003440.
Ly, K. H., Topooco, N., Cederlund, H., Wallin, A., Bergström, J., Molander, O., ... & Andersson, G. (2015). Smartphone-supported versus full behavioural activation for depression: a randomised controlled trial. PloS one, 10(5), e0126559.
Mak, W. W., Tong, A. C., Yip, S. Y., Lui, W. W., Chio, F. H., Chan, A. T., & Wong, C. C. (2018). Efficacy and moderation of mobile app–based programs for mindfulness-based training, self-compassion training, and cognitive behavioral psychoeducation on mental health: Randomized controlled noninferiority trial. JMIR mental health, 5(4), e8597.
Mantani, A., Kato, T., Furukawa, T. A., Horikoshi, M., Imai, H., Hiroe, T., ... & Kawanishi, N. (2017). Smartphone cognitive behavioral therapy as an adjunct to pharmacotherapy for refractory depression: randomized controlled trial. Journal of medical Internet research, 19(11), e373.
McCall, H. C., Hadjistavropoulos, H. D., & Sundström, C. R. F. (2021). Exploring the role of persuasive design in unguided Internet-delivered cognitive behavioral therapy for depression and anxiety among adults: systematic review, meta-analysis, and meta-regression. Journal of medical Internet research, 23(4), e26939.
McCloud, T., Jones, R., Lewis, G., Bell, V., & Tsakanikos, E. (2020). Effectiveness of a mobile app intervention for anxiety and depression symptoms in university students: randomized controlled trial. JMIR mHealth and uHealth, 8(7), e15418.
Meinlschmidt, G., Lee, J. H., Stalujanis, E., Belardi, A., Oh, M., Jung, E. K., ... & Tegethoff, M. (2016). Smartphone-based psychotherapeutic micro-interventions to improve mood in a real-world setting. Frontiers in psychology, 7, 1112.
Miller‐Graff, L., Ellis, K., & Hosny, N. (2021). PTSD coach online–Arabic: A randomized controlled pilot trial to examine feasibility, acceptability, and preliminary effectiveness. Journal of Traumatic Stress, 34(1), 23-34.
Miner, A., Kuhn, E., Hoffman, J. E., Owen, J. E., Ruzek, J. I., & Taylor, C. B. (2016). Feasibility, acceptability, and potential efficacy of the PTSD Coach app: A pilot randomized controlled trial with community trauma survivors. Psychological Trauma: Theory, Research, Practice, and Policy, 8(3), 384.
Moberg, C., Niles, A., & Beermann, D. (2019). Guided self-help works: randomized waitlist controlled trial of Pacifica, a mobile app integrating cognitive behavioral therapy and mindfulness for stress, anxiety, and depression. Journal of medical Internet research, 21(6), e12556.
Mulraney, M., Lee, C., Freed, G., Sawyer, M., Coghill, D., Sciberras, E., ... & Hiscock, H. (2021). How long and how much? Wait times and costs for initial private child mental health appointments. Journal of Paediatrics and Child Health, 57(4), 526-532.
Neumayr, C., Voderholzer, U., Tregarthen, J., & Schlegl, S. (2019). Improving aftercare with technology for anorexia nervosa after intensive inpatient treatment: A pilot randomized controlled trial with a therapist‐guided smartphone app. International Journal of Eating Disorders, 52(10), 1191-1201.
Newman, M. G., Jacobson, N. C., Rackoff, G. N., Bell, M. J., & Taylor, C. B. (2021). A randomized controlled trial of a smartphone-based application for the treatment of anxiety. Psychotherapy Research, 31(4), 443-454.
O’Dea, B., Han, J., Batterham, P. J., Achilles, M. R., Calear, A. L., Werner‐Seidler, A., ... & Christensen, H. (2020). A randomised controlled trial of a relationship‐focussed mobile phone application for improving adolescents’ mental health. Journal of Child Psychology and Psychiatry, 61(8), 899-913.
Oinas-Kukkonen, H., & Harjumaa, M. (2008). A systematic framework for designing and evaluating persuasive systems. In Persuasive Technology: Third International Conference, PERSUASIVE 2008, Oulu, Finland, June 4-6, 2008. Proceedings 3 (pp. 164-176). Springer Berlin Heidelberg.
Orosa-Duarte, Á., Mediavilla, R., Muñoz-Sanjose, A., Palao, Á., Garde, J., López-Herrero, V., ... & Rodríguez-Vega, B. (2021). Mindfulness-based mobile app reduces anxiety and increases self-compassion in healthcare students: a randomised controlled trial. Medical teacher, 43(6), 686-693.
Orji, R., & Moffatt, K. (2018). Persuasive technology for health and wellness: State-of-the-art and emerging trends. Health informatics journal, 24(1), 66-91.
O'Sullivan, S., et al. (2024). "A Novel Blended Transdiagnostic Intervention (eOrygen) for Youth Psychosis and Borderline Personality Disorder: Uncontrolled Single-Group Pilot Study." JMIR mental health 11: e49217.
O'Toole, M. S., Arendt, M. B., & Pedersen, C. M. (2019). Testing an app-assisted treatment for suicide prevention in a randomized controlled trial: effects on suicide risk and depression. Behavior therapy, 50(2), 421-429.
Pacella‐LaBarbara, M. L., Suffoletto, B. P., Kuhn, E., Germain, A., Jaramillo, S., Repine, M., & Callaway, C. W. (2020). A Pilot Randomized Controlled Trial of the PTSD Coach App Following Motor Vehicle Crash–related Injury. Academic emergency medicine, 27(11), 1126-1139.
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., ... & Moher, D. (2021). Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. Journal of clinical epidemiology, 134, 103-112.
Pensak, N. A., Carr, A. L., Jones, J., Mikulich‐Gilbertson, S. K., Kutner, J. S., Kilbourn, K., ... & Laudenslager, M. L. (2021). A pilot study of mobilized intervention to help caregivers of oncology patients manage distress. Psycho‐Oncology, 30(4), 520-528.
Raevuori, A., Vahlberg, T., Korhonen, T., Hilgert, O., Aittakumpu-Hyden, R., & Forman-Hoffman, V. (2021). A therapist-guided smartphone app for major depression in young adults: a randomized clinical trial. Journal of Affective Disorders, 286, 228-238.
Rodante, D. E., Kaplan, M. I., Olivera Fedi, R., Gagliesi, P., Pascali, A., José Quintero, P. S., ... & Daray, F. M. (2022). CALMA, a mobile health application, as an accessory to therapy for reduction of suicidal and non-suicidal self-injured behaviors: a pilot cluster randomized controlled trial. Archives of suicide research, 26(2), 801-818.
Roepke, A. M., Jaffee, S. R., Riffle, O. M., McGonigal, J., Broome, R., & Maxwell, B. (2015). Randomized controlled trial of SuperBetter, a smartphone-based/internet-based self-help tool to reduce depressive symptoms. Games for health journal, 4(3), 235-246.
Röhr, S., Jung, F. U., Pabst, A., Grochtdreis, T., Dams, J., Nagl, M., ... & Riedel-Heller, S. G. (2021). A self-help app for Syrian refugees with posttraumatic stress (Sanadak): randomized controlled trial. JMIR mHealth and uHealth, 9(1), e24807.
Roncero, M., Belloch, A., & Doron, G. (2019). Can brief, daily training using a mobile app help change maladaptive beliefs? Crossover randomized controlled trial. JMIR mHealth and uHealth, 7(2), e11443.
Roy, A., Hoge, E. A., Abrante, P., Druker, S., Liu, T., & Brewer, J. A. (2021). Clinical efficacy and psychological mechanisms of an app-based digital therapeutic for generalized anxiety disorder: randomized controlled trial. Journal of Medical Internet Research, 23(12), e26987.
Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford publications.
Saad, A., Bruno, D., Camara, B., D’Agostino, J., & Bolea-Alamanac, B. (2021). Self-directed technology-based therapeutic methods for adult patients receiving mental health services: systematic review. JMIR Mental Health, 8(11), e27404.
Sawyer, A., Kaim, A., Le, H. N., McDonald, D., Mittinty, M., Lynch, J., & Sawyer, M. (2019). The effectiveness of an app-based nurse-moderated program for new mothers with depression and parenting problems (eMums Plus): pragmatic randomized controlled trial. Journal of medical Internet research, 21(6), e13689.
Schlicker, S., Baumeister, H., Buntrock, C., Sander, L., Paganini, S., Lin, J., ... & Ebert, D. D. (2020). A web-and mobile-based intervention for comorbid, recurrent depression in patients with chronic back pain on sick leave (Get. Back): pilot randomized controlled trial on feasibility, user satisfaction, and effectiveness. JMIR Mental Health, 7(4), e16398.
Schlosser, D. A., Campellone, T. R., Truong, B., Etter, K., Vergani, S., Komaiko, K., & Vinogradov, S. (2018). Efficacy of PRIME, a mobile app intervention designed to improve motivation in young people with schizophrenia. Schizophrenia bulletin, 44(5), 1010-1020.
Schueller, S. M., Muñoz, R. F., & Mohr, D. C. (2013). Realizing the potential of behavioral intervention technologies. Current Directions in Psychological Science, 22(6), 478-483.
Stallman, H. M. (2019). Efficacy of the My Coping Plan mobile application in reducing distress: A randomised controlled trial. Clinical Psychologist, 23(3), 206-212.
Steare, T., O’Hanlon, P., Eskinazi, M., Osborn, D., Lloyd-Evans, B., Jones, R., ... & Johnson, S. (2020). Smartphone-delivered self-management for first-episode psychosis: the ARIES feasibility randomised controlled trial. BMJ open, 10(8), e034927.
Stiles-Shields, C., Montague, E., Kwasny, M. J., & Mohr, D. C. (2019). Behavioral and cognitive intervention strategies delivered via coached apps for depression: Pilot trial. Psychological services, 16(2), 233.
Stolz, T., Schulz, A., Krieger, T., Vincent, A., Urech, A., Moser, C., ... & Berger, T. (2018). A mobile app for social anxiety disorder: A three-arm randomized controlled trial comparing mobile and PC-based guided self-help interventions. Journal of consulting and clinical psychology, 86(6), 493.
Subotic-Kerry, M., Borchard, T., Parker, B., Li, S. H., Choi, J., Long, E. V., ... & O’Dea, B. (2023). While they wait: A cross-sectional survey on wait times for mental health treatment for anxiety and depression for Australian adolescents. Medrxiv, 2023-08.
Sun, Y., Li, Y., Wang, J., Chen, Q., Bazzano, A. N., & Cao, F. (2021). Effectiveness of smartphone-based mindfulness training on maternal perinatal depression: randomized controlled trial. Journal of Medical Internet Research, 23(1), e23410.
Tønning, M. L., Faurholt-Jepsen, M., Frost, M., Martiny, K., Tuxen, N., Rosenberg, N., ... & Kessing, L. V. (2021). The effect of smartphone-based monitoring and treatment on the rate and duration of psychiatric readmission in patients with unipolar depressive disorder: the RADMIS randomized controlled trial. Journal of Affective Disorders, 282, 354-363.
Torok, M., Han, J., McGillivray, L., Wong, Q., Werner-Seidler, A., O’Dea, B., ... & Christensen, H. (2022). The effect of a therapeutic smartphone application on suicidal ideation in young adults: Findings from a randomized controlled trial in Australia. PLoS Medicine, 19(5), e1003978.
Torous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018). Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. BMJ Ment Health, 21(3), 116-119.
Torous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018). Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. BMJ Ment Health, 21(3), 116-119.
Tregarthen, J., Kim, J. P., Sadeh-Sharvit, S., Neri, E., Welch, H., & Lock, J. (2019). Comparing a tailored self-help mobile app with a standard self-monitoring app for the treatment of eating disorder symptoms: Randomized controlled trial. JMIR Mental Health, 6(11), e14972.
Valentine, L., McEnery, C., Bell, I., O'Sullivan, S., Pryor, I., Gleeson, J., ... & Alvarez-Jimenez, M. (2020). Blended digital and face-to-face care for first-episode psychosis treatment in young people: qualitative study. JMIR Mental Health, 7(7), e18990.
van Aubel, E., Bakker, J. M., Batink, T., Michielse, S., Goossens, L., Lange, I., ... & Myin-Germeys, I. (2020). Blended care in the treatment of subthreshold symptoms of depression and psychosis in emerging adults: A randomised controlled trial of Acceptance and Commitment Therapy in Daily-Life (ACT-DL). Behaviour Research and Therapy, 128, 103592.
van der Meer, C. A., Bakker, A., van Zuiden, M., Lok, A., & Olff, M. (2020). Help in hand after traumatic events: A randomized controlled trial in health care professionals on the efficacy, usability, and user satisfaction of a self-help app to reduce trauma-related symptoms. European Journal of Psychotraumatology, 11(1), 1717155.
Welles, C. C., Tong, A., Brereton, E., Steiner, J. F., Wynia, M. K., Powe, N. R., ... & Cervantes, L. (2023). Sources of clinician burnout in providing care for underserved patients in a safety-net healthcare system. Journal of General Internal Medicine, 38(6), 1468-1475.
Wilhelm, S., Weingarden, H., Greenberg, J. L., Hoeppner, S. S., Snorrason, I., Bernstein, E. E., ... & Harrison, O. T. (2022). Efficacy of app-based cognitive behavioral therapy for body dysmorphic disorder with coach support: initial randomized controlled clinical trial. Psychotherapy and psychosomatics, 91(4), 277-285.
Wong, V. W. H., Ho, F. Y. Y., Shi, N. K., Tong, J. T. Y., Chung, K. F., Yeung, W. F., ... & Sarris, J. (2021). Smartphone-delivered multicomponent lifestyle medicine intervention for depressive symptoms: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 89(12), 970.
Wu, A., Scult, M.A., Barnes, E.D. et al. Smartphone apps for depression and anxiety: a systematic review and meta-analysis of techniques to increase engagement. npj Digit. Med. 4, 20 (2021). https://doi.org/10.1038/s41746-021-00386-8
Yang, E., Schamber, E., Meyer, R. M., & Gold, J. I. (2018). Happier healers: randomized controlled trial of mobile mindfulness for stress management. The Journal of Alternative and Complementary Medicine, 24(5), 505-513.
Yang, Y. J., & Chung, K. M. (2023). Pilot randomized control trial of an app-based CBT program for reducing anxiety in individuals with ASD without intellectual disability. Journal of Autism and Developmental Disorders, 53(4), 1331-1346.

No competing interests reported.

PRISMA2020checklistLV.docx

Download PDF

Reviewers agreed at journal
11 Nov, 2024
Reviewers invited by journal
08 Nov, 2024
Editor assigned by journal
27 Oct, 2024
Submission checks completed at journal
24 Oct, 2024
First submitted to journal
21 Oct, 2024

You are reading this latest preprint version

Systematic Review and Meta-Analyses of Mental Health App RCTs: Persuasive Design, Engagement, and Efficacy

Status:

Version 1

Abstract

Figures

Background

Methods

Eligibility Criteria

Synthesis Approach

Search Strategy

Study Selection & Data Collection

Data Extraction

Risk of Bias assessment of individual studies

Meta-Analysis, Meta-Regression, and Correlation Analysis

Results

Included studies

Study Characteristics

Risk of Bias

Publication Bias

1. Efficacy

Meta-Analysis: Overall and Sub-Group Effects of Intervention Efficacy

2. Persuasive Design Principles

Meta-Regression: Persuasive Design Principles and Intervention Efficacy

Persuasive Design Principles and Engagement

Discussion

Limitations

Conclusion

Declarations

Competing Interests

Author Contribution

Acknowledgments

Data Availability

References

Additional Declarations

Supplementary Files

Status:

Version 1