Understanding how engagement is reported in the RMT literature is imperative in order to ensure reproducibility in the field, allow studies to build upon results of previous work, and to implement findings in real world settings. The first and second aims of this review were to explore the current state of defining and measuring engagement by using both quantitative and CIS data synthesis methods. The majority of papers reported on engagement in some form. However, these represented a large range of incoherent and often unjustified measures. Many studies employed several measures in one paper, resulting in a lack of distinction between measure types, explanation of why some were chosen over others, and understanding of what conclusions might be drawn from the engagement findings. A much lower proportion of studies included a corresponding definition of engagement, including many of those which described engagement as a main aim of the study. Where engagement was defined, concepts and phrases were interchangeable across papers. Indeed, even across papers with similar engagement definitions, such as ‘feasibility’, cut-offs used to measure RMT compliance differed hugely. Thus, though there is potential for the field to evaluate engagement, a lack of standardised reporting is impeding progress.
The third aim of this review was to present recommendations for the standardization of future work. The first step towards this has been to provide clarity on the engagement measures that are currently used in the literature. The integrative framework depicted in Fig. 2 is split into two core themes of engagement measures: engagement with the research protocol and engagement with RMTs themselves. This distinction is important given that many studies did not differentiate between these concepts when reporting on engagement; using correlates of dropout from RMT studies as a proxy for gauging wider interest in RMT implementation is only relevant if the reasons for dropout were specific to the RMT aspects of the study. The engagement with RMTs section is further split into objective and subjective measurements. Interestingly, a wide range of measures were used here, but very few studies acknowledged a distinction between objective and subjective engagement or the possible interactions between the two. Under each section of the framework lies a series of options for measuring each engagement type. This framework should aid in the classification and selection of measurements when reporting on an RMT study.
The second step towards standardising future work lies in establishing ‘best practice’ guidelines for reporting on engagement. This review has highlighted several drawbacks in the current literature: a dearth of clearly conceptualised engagement definitions and corresponding measures, and, as a result, an inability to make concrete conclusions on the extent of engagement in the field. Figure 3 depicts how the process of assessing engagement with RMTs should begin as early as study development. Authors designing RMT studies are encouraged to explore the reasons for examining engagement, e.g. understanding the feasibility of using a symptom tracking app, and why it is important to know this, e.g. in order to understand the extent of data collection that could be expected with clinical implementation, or to inform approaches to missing data. Authors should then pre-define a definition, measurement(s) and applicable cut-offs to be used in answering this question. Such data could be reported in the main paper or a subsequent engagement paper. Following these guidelines could provide the necessary foundations for reproducible results in the field.
[Insert Fig. 3]
Links with previous work
To our knowledge, this is the first review that has explored the extent of engagement reporting across the entirety of the RMT literature. This work complements and expands on previous reviews that have found heterogeneity in reporting of RMT studies 22–24. It also might provide explanation as to why previous studies have found a wide range of dropout rates 20,21. A lack of provision of engagement definitions in this work reflects the findings of Sieverink and colleagues in the eHealth literature 27; a small minority of studies included an operationalization of the engagement measure used, making comparisons difficult at best and futile at worst.
A key link to previous work is the parallel between the objective and subjective engagement concepts highlighted in Perski and colleagues’ work with DBCIs 28. Both the definition and measurement synthetic constructs found in this review loosely map on to this distinction, with two exceptions. First, subjective engagement with RMTs is largely focused on the usability or utility of the technology, or on the subjective effects of tracking symptoms, e.g. self-awareness of symptoms, feeling monitored by a ‘safety net’. This is in comparison to the typical measures of ‘flow’ or immersion that are seen as a mechanism of action towards behavior change in DBCIs. Second, engagement with the research protocol is a novel finding that has not been acknowledged in the DBCI literature. This is likely owing to the simultaneous use of RMTs as both a method for data collection and a tool for symptom self-management. Such differences warrant the exploration of the RMT field in its own right, as separate from DBCI or general eHealth literature.
Strengths and limitations
This review was deliberately extensive in nature, including papers spanning physical and mental health conditions and aRMT or multiparametric RMT measures. It did not exclude papers for not reporting on engagement. This allowed for an overview of the scope of reporting across the field, in contrast to previous reviews which pre-defined engagement for the purpose of inclusion 20,22,27,88. However, one limitation of such a broad focus is the lack of sub-typing by condition, journal type, RMT type or RMT purpose. For example, objective RMT engagement might be more important to consider in studies using RMTs for research purposes, whereas subjective RMT engagement might be more insightful for RMTs implemented into clinical practice. A specific focus on pRMT studies might also uncover additional measures of objective and subjective engagement. A second limitation is the decision to include secondary analyses papers in data synthesis. Six of the included papers reviewed analyses from the MONARCA I trial 43,45,46,51,58,89, and a subsequent 5 on the MONARCA II trial 64,66,85,87,90. This was justified given the tendency to report on feasibility in standalone papers, however, may have resulted in an over-representation of definitions or measures chosen by these authors. It might also be the case that papers reporting on the same dataset publish multiple analysis papers but only one engagement paper, resulting in a skew towards measures of engagement with the research protocol in these findings. It should also be considered that the searches for this review were undertaken in July 2020. However, the authors have no reason to believe that engagement reporting standards have radically changed between this date and publication of this manuscript.
Implications for future work and conclusions
This review provides the foundation for future RMT studies to define, measure and report on engagement in a standardized and reproducible way. Authors should use both the engagement reporting process (Fig. 3) and the integrative framework (Fig. 2) as a basis for assessing engagement. At the same time, further work should be undertaken to understand how engagement reporting might differ by condition, journal type, RMT type and RMT purpose. Such findings will build on the proposed framework, making it applicable to a wider range of RMT work as the field continues to grow.
This review suggests that where authors are generally interested in measuring engagement with RMTs, the primary emphasis is placed on engaging with the research protocol or objectively with the RMTs themselves. This is logical for two reasons. First, the CONSORT-EHEALTH guidelines 25 focus mainly on reporting of engagement with the research protocol, e.g. participant attrition, and technology process outcomes, e.g. metrics of use. Second, these types of engagement directly impact on missing data and resulting statistical analyses. However, a deeper exploration of subjective engagement with RMTs might aid further understanding. A handful of papers in this review have already examined the effects of monitoring symptoms remotely on both clinical outcome variables and feelings of self-awareness and safety. A few have also begun to acknowledge the interaction between RMT usability and future use. Future work should focus on uncovering the links between objective and subjective RMT engagement, which could have huge implications for understanding how and why users engage with these technologies for research, clinical practice, or self-management.
To conclude, the current review provides an exploration of the current state of engagement reporting in studies which use RMTs for symptom tracking in physical or mental health conditions. Where there is clearly interest, the growth of the field is currently impeded by a lack of engagement definitions, incoherent measures, and an inability to compare findings. Recommendations for the standardization of reporting guidelines and the integration of engagement definitions and measures into the study design process have been put forward. In extending existing reporting guidelines, engagement with RMTs should also be considered as distinct from general eHealth interventions.
Future work should aim to contribute to the ongoing framework that has been provided, moving towards a unified understanding of the impact of engagement in this field, and what can be done to promote it.