Towards Realist-informed Ripple Effects Mapping: Positioning the approach

doi:10.21203/rs.3.rs-3943638/v1

Download PDF

Research Article

Towards Realist-informed Ripple Effects Mapping: Positioning the approach

https://doi.org/10.21203/rs.3.rs-3943638/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 30 Oct, 2024

Read the published version in BMC Medical Research Methodology →

You are reading this latest preprint version

Background: Evaluation approaches such as ripple effects mapping (REM) and realist evaluation have emerged as popular methodologies to evidence impact, and the processes of change within public health as part of whole systems approaches. Despite the various examples of their implementation across different evaluation settings, there has been little or no evidence of how they might be effective when combined.

Methods: With REM’s potential to pragmatically illustrate impact, and realist evaluation’s strength to identify how and why impacts emerge, this paper develops a rationale and process for their amalgamation. Following this, we outline a realist-informed ripple effects mapping (RREM) protocol that may be suitable for application within evaluation settings in a range of public health, whole system and physical activity settings.

Discussion: Combining these two approaches has the potential to more effectively illuminate the impacts that we see within public health and whole system approaches and initiatives. What is more, given the complexity often imbued within these approaches and initiatives they hold capability for also capturing the causal mechanisms that explain these impacts.

Conclusions: It is our conclusion that when combined, this novel approach may help to inspire future research as well as more effective evaluation of public health and whole system approaches. This is crucial if we are to foster a culture for learning, refinement and reflection.

Realist

RREM

Causal connections

Ripple Effects Mapping

Seldom do programmes, policies, interventions or whole systems approaches, which are (whole of systems approaches being cross sector collaborations to address social / structural challenges at a combination of levels (1), work exactly as intended within specific contexts. For those involved in evaluating these initiatives and approaches, this unpredictability poses a significant challenge given the multiplicity of impacts occurring, the channels by which these impacts come about, and the circumstances in which they happen. Simultaneously, those who invest expect certain levels of accountability and evidence for what happens.

Therefore, it is important for evaluations to capture information not only on what was anticipated to happen, but also to understand what happens in reality. There are several methods available to researchers who wish to evaluate whole system approaches (1, 2) which are implemented within complex adaptive systems. Furthermore, we are seeing more calls for complexity rich methodologies, because it is not always possible to provide simple answers to whole system approaches and research questions (1, 3). Ripple effects mapping (REM) and realist evaluation are two such established evaluation methods (4, 5, 6, 7, 8) that are gaining traction. In this paper, we provide a rationale and protocol for bringing these two approaches together to form realist-informed ripple effects mapping (RREM). Throughout this paper, we position this rationale and protocol within the following aims and objectives: 1. articulating the justification for merging the two approaches, 2. illustrating the process (the protocol) for carrying out RREM, and 3. providing insights and discussion for what this integration means and possible future directions. To strengthen these three aims and the protocol, we provide a supporting case study (see appendix) that utilised a RREM protocol to evaluate a whole systems asset-based community development approach within a physical activity organisation in the UK.

In essence, the central premise of our paper is one that calls for pragmatic innovation in evaluation practice that draws upon the strengths of exploring the merging of different evaluation approaches (such as REM and realist evaluation). This is opposed to being grappled in methodological and philosophical debates about the tensions in their convergence (however we do acknowledge that there should always be critical space to discuss differences in methodological foundations). This stance is supported by Barbrook-Johnson et al., (9), who have called for more practical methods of evaluation that could aid the exploration of complexity. And in doing so, they highlight the following point;

“Complexity-appropriate evaluation methods do not have to be sophisticated or highly technical. Often the best strategy is to co-produce and customise together with users and stakeholders a combination or hybrid of existing methods, that are adaptable, iterative and appropriate. Many methods can be repurposed for complex policies and contexts. The innovation is getting them in the right place, in the right hands, and using them in the right combination, at the right time” (9: pg.13).

REM

REM is a qualitative method of capturing the wider intended and unintended impacts of an initiative. The participants of a REM workshop include stakeholders that are involved in, or affected by, the whole system approach or initiative, ranging from those involved in the funding and strategy, to individuals that are affected (for example local residents who participate in a physical activity initiative run by the local authority). These stakeholders are invited to one or more participatory workshops. The workshops aim to create a visual depiction of the activities that have occurred within a given timeframe, the impacts that have occurred, and then the secondary or tertiary impacts which followed thereafter (i.e., the ripple effects). Importantly, REM focuses on understanding the interconnections between these activities and impacts.

There are several different approaches to conducting REM, as outlined in the field guide by Chazdon et al., (10), from web mapping to in-depth rippling, to theming and rippling. Nobles et al., (5) recently advanced the in-depth rippling technique by both using REM concurrent to the implementation of an initiative (rather than post-implementation) and placing more attention on the temporal nature of the activities and impacts (through use of a timeline). This version of REM has gained significant traction over the last 2–3 years in the public health context – e.g. physical activity (11, 12), obesity prevention (13), sexual health (14), and workplace health (15).

Realist Evaluation

Realist evaluation makes use of qualitative and quantitative methods to understand how, why and for whom something works (16). Realist evaluation is rooted in, and informed by, the philosophical realms of realism; in short, realist evaluators accept that there is a reality out there independent of our knowledge of it (16, 17). To understand how and why changes occur, evaluators need to make sense of the generative causation at play in context; by this we mean the hidden mechanisms of change (18) that influence outcomes that will vary across different settings (contexts) (19). Indeed, these contextual nuances will influence the outcomes we see.

Realist evaluation is usually underpinned by three interdependent phases. The first phase centres around the development of programme theories (i.e. explanatory statements) to hypothesise how, for example an initiative works, often referred to as contexts, mechanism reasoning, mechanism resources and outcomes (see 18). The second phase involves the testing and exploration of these theories via mixed methods data collection (20). The final phase focuses on the refinement of the theories following analysis to better describe how and why the initiative works as it does.

Bringing the ‘Real’ into REM

Both REM and realist evaluation have unique strengths. REM is a pragmatic method, which can actively engage and captivate a wide range of stakeholders. Data collection is relatively simple and can be adapted based on the resources available (12). As a result, REM can quickly build a high-level account of what has occurred within an initiative or whole systems approach over a period of time. REM does not however surface the generative causation which is so vitally important in understanding whole system approaches and how they operate within real world systems. Conversely, realist evaluation can explore what the key mechanisms are for generating change, enabling us to explain how and whyimpacts emerge. However, unlike REM one of the challenges facing realist evaluation is the degree of stakeholder accessibility and participation; sometimes attributed to the philosophical complexity of the approach (21, 22, 23), and the conceptual jargon often underpinning it (24). Thus, there is a strong justification for drawing on the strengths of each approach, (REM and realist evaluation), and bringing these together to form RREM.

Research suggests that both REM and realist evaluation are approaches that are open to methodological innovation and wider exploration. Realist evaluation has already seen a wide range of scholars experiment and explore the fusion of the approach with other approaches such as Q-Methodology (25, 26, 27, 28), soft systems approaches (29) and economic evaluation (30). This places us in a position of openness and innovation to further enhance these valuable evaluation methodologies. The following table positions REM and realist evaluation next to each other identifying some of the strengths and limitations of the approaches. The idea of convergence seeks to show the complementary nature of the two for strengthening the approaches and mitigating some of the limitations.

Table 1 near here

Table 1

*Aligning the principles of REM and realist evaluation.*
Principles of REM	Principles of Realist Evaluation
Qualitative method that can allow us to capture the impacts of an initiative or whole systems approach over time.	Theory driven approach that can encompass mixed methods to understand the generative causality of the impacts.
Identifies the ‘what’. is going on.	Explores the how and why behind the ‘what’.
Participatory and engaging process for stakeholders.	Often limited stakeholder accessibility and participation.
Straight forward data collection method.	Philosophically complex approach. Data collection could be cumbersome based on various methods used.
Can capture changes over time with stakeholders.	May capture generative changes over time sensitive to different contexts, but may limit engagement with wider stakeholders.
Towards Convergence: RREM
• An iterative evaluation approach, applicable for use in whole of system approaches. • An approach that brings complexity driven principles into pragmatic and participatory settings. • Accessible and engaging for stakeholders, using a participatory method of collecting data. • Key focus on identifying, investigating and learning from impacts as a result of a programme, policies or intervention. • Opportunity to develop ‘causal connections’ within and behind impact pathways. • Variety of outputs including visual maps, qualitative and quantitative data.

Table 2

*Example of PREM workshop structure.*
Activity	Prompts to help participants
Introduction and warm up
The facilitator(s) should introduce the RREM workshops to the participants and initiate the warm up activity. The activity can be innovative in a way that encourages conversation and reflection, using questions influenced by appreciative inquiry, although it is vital to consider challenges and things that have not gone well.	• What is important to you? • What is important to your organisation? • What motivated you to contribute to the area of focus? • What would be the ideal outcome of your project? • What have the positive outcomes been? • What have been some of the challenges? • What partnerships played an important role?
Mapping activity
Mapping the impacts The facilitator(s) works with participants to confirm a timeline for the mapping. The participants will then map the impacts they identify and link them to other impacts.	• What are the impacts that participants have observed or experienced? • Were there any additional impacts as a result? • Focus on the impacts. • Estimate the time if you are not sure. • It is fine to work forwards or backwards from an impact.
2. Adding extra detail Participants are encouraged to add additional detail that provides additional context. This can also explore the causal connections between the impacts.	• Can you highlight what impacts were unintended? • Can you identify who has been impacted and why? Were these intended or unintended? • Were there any financial impacts? • Are there wider contextual factors that are important to these impacts? • What do you predict will happen in the next 3 months?
3. Most and least significant impact pathways Participants should select pathways based on the perceived impact in relation to the time and resource committed to the pathway. The researcher should use this opportunity to dig deeper with questioning.	• Can you highlight the 2 most significant impact pathways in your view? • Can you highlight the 2 least significant impact pathways based on the resource and time invested into the pathway? • Why have you selected these pathways?
Reflections and next steps
The facilitator(s) should use this opportunity to allow participants to reflect and share thoughts from their participation. The facilitator(s) should also share the next steps so the participants are clear and able to ask any questions.	• What are your reflections from the workshop? • Were there any impacts that surprised you? Can you tell us why?

From the outset we refer to this protocol as a RREM methodology. We present an iterative evaluation methodology that brings together the key strengths of the two approaches, which can be applied to any setting where it is important to understand impacts and the explanations of change that lead to them in whole system approaches. The RREM methodology does not favour REM or realist evaluation as the dominant methodology, and what we present is a synthesis of the two (see Fig. 1). As a starting point, we situate the traditional process of REM (for us, we refer to the Nobles et al., (5) application of REM) into a realist evaluation framework which commonly comprises the phases of developing programme theory, testing programme theory and refining programme theory. For greater accessibility, we re-frame these phases as:

Phase 1. Identifying impacts and their perceived explanation.

Phase 2. Investigating impacts and their causal connections.

Phase 3. Learning from our impacts and their causes to inform practice.

Figure 1 near here

The protocol that we present below is designed to suit a wide range of public health / whole system approaches and / or initiatives. Additionally, there is no prescription regarding how long the RREM implementation process lasts. The length of each phase should be determined relative to the context of the evaluation and its aims and objectives. Within this paper we have produced an appendix that details a RREM evaluation (across the three phases) that the authors were engaged in within Active Essex (A physical activity organisation in the UK) spanning twelve months. We provide this as a supplementary resource to this paper that may bring to life the reality of the process across the three phases.

Phase 1: Identifying Impacts and their Perceived Explanation

Figure 2 near here

In the first phase of RREM, the foundational stages of a realist evaluation are brought to life where the evaluation team works with key stakeholders of stakeholders including those who represent communities regarding a specific whole system approach or initiative to identify the underlying theories and explanations behind the outcomes and impacts they believe they are seeing in their work (see Fig. 2). Here, we use outcomes and impacts interchangeably on the basis that they capture the key results and contributions that are seen within a programme or system. Secondly, we deploy these key principles within the participatory and inclusive nature of the REM process (5) to encourage co-production.

Discovery Workshops

This phase begins with ‘Discovery workshops’ that bring together stakeholders involved in, or affected by, the whole system approach or initiative. The purpose of this workshop is to confirm the aims and objectives of the evaluation, which helps to prevent stakeholders and researchers moving too quickly into the active evaluation without having clear questions, objectives, and boundaries. Discovery workshops enable the evaluation team to establish the focus of the inquiry, by asking:

What is important to stakeholders, and what do they think is important for exploration? For example, if looking at a broad topic, are there specific areas that should be looked at and where may there be impacts emerging?

Is there motivation for collaboration (31) to encourage stakeholders to take agency in the evaluation? This may involve developing mutual professional relationships (31).

What evidence is currently available to inform what is evaluated? Identification of these sources may highlight issues that have already been understood which may avoid duplication.

What is feasible and achievable for the evaluation? This requires open and honest conversations about what is practical and realistic whilst remaining focused on the aims and objectives of the evaluation.

In keeping with the participatory nature of REM the discovery workshop is important for involving a range of stakeholders invested in this work, including those who represent communities and the whole system approach under exploration. This process can be conducted in person or virtually using software such as Miro, Microsoft PowerPoint, Canva, or Vensim to visually illustrate the key themes/topics identified as important by the group. Within the discovery workshops, participants are asked to consider how and why these areas of focus are important because impacts and outcomes will naturally emerge in discussion, which the evaluation team can make note of for future exploration.

RREM 1: Mapping provisional impacts and identifying theories to test.

The first RREM workshop aims to identify impacts associated with the delivery of the initiative. The workshop follows the general principles as presented in Nobles et al., (5). Typically, a workshop will last 2.5 to 3 hours and can be hosted either in person or online with as many as 15–20 participants working on multiple maps. Here, participants should be split into groups that can contribute to the relative maps, confirmed at the discovery workshop, and be given the opportunity to tell their story, discussing the successes and challenges experienced or observed within the area of focus. This can be supported by the facilitator by using questions influenced by appreciative inquiry (5).

The workshop then encourages participants to map activities (or actions), impacts and ripple effects (i.e., those that occur because of another impact). The links between activities and impacts are key in identifying where impacts happen as a result of other impacts and are referred to as ‘impact pathways’ (multiple impacts that are linked together within the RREM workshops), see Nobles et al., (5). However, what is different within the RREM approach is that participants are encouraged to dig deeper into the links identified between impacts. Facilitators prompt participants to explore the links by asking ‘how?’, ‘why?’ and ‘for whom?’ to understand the causality underpinning the link between activities and impacts, and what it is about context that influences these. This happens in real time during the workshop and is captured to inform the next stage of the process. Before the workshop is concluded, it is important for participants to highlight the most and least significant impact pathways that should be considered for further testing.

Post RREM Workshop 1

After the workshop, it is useful to digitise the maps produced in the workshop utilising the software previously highlighted. As part of this production, it may be necessary to add to the map through the process of corroboration by gathering additional data from key stakeholders who were not able to attend (in addition to those who attended). These may be supported by one-to-one ‘How and Why’ conversations (inspired by Manzano’s (32) paper on crafting realist interviews) that start to uncover how and why the observed impacts and changes are occurring. This process helps to establish the impact pathways identified within the workshop, and provides opportunity for further exploration, which we discuss in the next section.

Developing Causal Connections

Having identified the impact pathways to focus on, the research team develop ‘causal connections’ that illustrate the deeper causality underpinning these impact pathways. These causal connections represent the realist causal impact pathways that within a realist ontology (33), sit behind and beneath the traditional REM impact pathways. These causal connections are developed using realist heuristics such as context mechanism resource, mechanism reasoning and outcomes (18), or explanatory statements to uncover the generative causality behind how and why impacts are generated. Using a retroductive approach that theorises and tests hidden mechanisms (33), researchers are able to use their inferred knowledge and understanding (from the discovery workshop, RREM workshop 1 and grey literature) of the evaluation topic to make explicit the key reasons about how and why an impact has or has not occurred. This process of causal connectivity is particularly useful for ‘what impacts do we expect to see in the upcoming months’ part of the mapping (usually, the next three months is proposed as a suitable time period (5). In our application of RREM, our causal connections are drawn up through the explanatory process of “IF… THEN… BECAUSE…” statements (34). This format has been chosen to ensure that the causal connections are digestible for all participants to avoid disengaging people with realist jargon (please see appendix for specific example).

At this stage, the causal connections identified could be linked to literature to identify potential underpinning theories (see Greenhalgh and Papoutsi (35)) that represent the area of focus (see appendix for example). Overall, this whole process culminates in the production of a RREM map that is underpinned with initial causal connections ready for testing and exploration in phase 2.

At the end of the RREM phase 1, the following actions will have taken place:

Discovery workshops
RREM workshop 1
Digitising map and sharing with participants
Identifying causal connections

Phase 2: Investigating Impacts and Their Causal Connections

Figure 3 near here

Phase 2 involves the explicit investigation of the impact pathways and causal connections that were identified in phase 1. Within the realist approach, this is a testing and exploratory phase using data from RREM workshop one and two alongside a variety of additional qualitative and quantitative methods to explore how, why and for whom these pathways are manifesting.

RREM 2 workshop: Testing and Refining Impact Pathways and Causal Connections

Prior to delivering the second RREM workshop, a period of time (usually a three-month gap (5) is recommended to encourage further activities and impacts to occur (however, this gap could be longer, or even shorter). This is also important to allow the ‘what impacts do we foresee happening in the coming months’ to take shape. In essence, the space between workshop 1 and workshop 2 provides the whole system approach or initiative with time to ‘live’ and ‘breathe’ within its unique context. After the appropriate amount of time has elapsed, participants are gathered together once again for the second RREM workshop.

Within the second workshop (and as depicted within the phase 2 graphic, see Fig. 3), the completed maps from phase 1 are presented to participants with the intention to create participation, inclusion and critical engagement. The most significant impact pathways are also illustrated and explained at this stage. In addition, the causal connections that lie beneath, within and around the impact pathways are also explained to participants. For many in attendance at this workshop, it will be their first sight of the maps, and in particular, the causal connections that have been developed. It is here where they have the opportunity to reflect upon the map, make any changes to what is being presented and to refine accordingly.

Following on, participants reflect upon what has happened over the last three months making changes to the impact pathways and discussing the causal connections that were developed as part of a testing process Participants have the freedom to critically discuss these causal connections, which involves the process of adapting and refining them ready for the next iteration of RREM. This process also includes the introduction of rival causal connections or theories, which further encourage the participants to respond in more depth. This is a unique realist contribution to the RREM process in that theories are being tested in a collaborative manner. This is seldom achieved within a realist evaluation as participants do not always have an explicit role in testing theory; rather, they are usually products of theory testing. As such, both the participants and the research team co-produce the process of theory testing and refinement.

The role of the research team in this workshop is to stimulate evaluative consciousness, using questioning influenced by the identified causal connections, wherein participants start to think critically about their activities and impacts (31), and consider the strengths and weaknesses of what has occurred. In addition and in accordance with a knowledge exchange environment (36, 37), participants are also bringing their own critical offerings to this learning environment which in turn fosters learning within the research team. The key output from this workshop is an adapted REM map that shows development, growth, and reflexivity in regard to the impact pathways and respective causal connections.

Further Testing and Investigation of Impact Pathways and Causal Connections

After the RREM 2 workshop, an additional phase of testing and exploration takes place. Here, the research team operationalise a series of realist-informed qualitative and quantitative methods to further test the impact pathways and causal connections to explore how, why and for whom the pathways apply. In accordance with realist evaluation, a methods neutral approach is taken (20) where those methods that are most suited to testing the impact pathways and causal connections are selected. These methods may consist of surveys that provide more extensive reach to test the impacts on a wider sample. For example, in a community development setting, this may be valuable in terms of accessing key members who did not attend the mapping workshop but are still given an opportunity to inform the development of the maps. The same can also be said for implementing qualitative methods of interviews and focus groups (32). In particular, being able to have deeper ‘How and Why’ conversations with people who may have, or not, been at the workshops helps to further investigate the impact pathways and accompanying causal connections. There is also the potential here to experiment with other qualitative methods focusing on vignettes or stories of change (38) that represent and bring to life some of these impacts.

Essentially, this further investigation is the explicit testing of the impact pathways and causal connections that will be shaped for presentation and sense making in the third phase of RREM. This additional wave of testing helps to bring more value and credibility to workshop findings that might not always reflect the true reality of what is happening in practice. This use of additional and complementary methods has been advocated for by REM methodologists (12).

Analysing and Synthesising Findings

Having mobilised the selected methods to support the testing, this stage of phase 2 involves the research team familiarising themselves with the data, and then moving deeper into analysis. As Nobles et al., (12) state, there is no prescribed approach for how impact pathways are analysed in REM. The analytic approach may adopt an inductive or deductive (10, 12) content analysis that draws upon the data presented in the maps. However, within a RREM approach there are multiple sources of data that require analysis and integration that draw upon the maps as well as additional qualitative and quantitative methods. It is here where we suggest that a realist inspired retroductive position (33) is drawn upon, whereby the research team searches for generative causal insights that lie beneath the surface of what can actually be seen. For example, the CMMO (18) heuristic may assist with coding contexts, reasoning and reactions that are at play within the impact pathways and connections under test. This iterative process of data analysis and integration results in the provisional refinements to the RREM map and respective impact pathways and their causal connections. This process culminates in a rich data set that informs the illustration of a refined set of impact pathways and underlying causal connections ready for presentation, deliberation and learning for phase 3.

At the end of the RREM phase 2 the research team will have fulfilled the following tasks:

Prepared and delivered RREM workshop two.

Used RREM workshop 2 to present impact pathways and supporting causal connections to participants for amendment and refinement.

Mobilised a series of additional methods for testing post workshop 2.

Analysed data and synthesised findings for presentation in phase 3.

Phase 3: Learning From our Impacts and Their Causes to inform practice

Figure 4 near here

As we move into the third and final phase of RREM, by this point an additional period of time has elapsed to take into account the testing of the theories alongside the additional methods of exploration. In line with phases one and two, there is no firm prescription on the duration of time between phase two and three which should be guided by the aims and objectives of the evaluation, and the resources available. In this final phase of the RREM process the research team operationalise key stages of learning and refinement (see Fig. 4). This idea of learning and refinement is an important point of alignment for REM and realist evaluation because both approaches encourage this endeavour. For example, REM promotes the appreciative inquiry approach that encourages participants within a co-design environment to ‘discover, define, dream and design’ approaches that have had a positive impact (39). And, within a realist position, the programme theory refinement stage encourages stakeholders to make sense of the theories that have been tested, and what their refinements look like. In turn, Shulha et al., (31) suggests that whole system approach or initiative stakeholders consider what these refinements mean for informing use and action in initiative design.

RREM Workshop 3: Presenting and Refining Learning

The final workshop aims to present findings that represent the refined causal connections and impact pathways. In terms of participation, it is of value to involve all stakeholders consistently across the phases in each workshop, while also establishing routes or engagement with stakeholders who are unable to attend all of them. In practical terms, the research team brings to life a near-finalised RREM map which also includes suggested refinements to the impact pathways and causal connections. At this stage the research team has also the opportunity to present crafted “stories of change” (38) to illustrate the impact pathways and causal connections identified (refer to appendix for an example). In this workshop, rather than accepting them as a given, these findings are presented in a provisional sense that allow for corroboration on the part of the participants to discuss and refine what is presented in the maps. Below, we list some of the questions that guide this workshop:

What are the impacts that remain valid and what are the explanations behind them?
What are the suggested causal connection refinements and what role do participants have in inclusively agreeing them and shaping them?
Can agreement be established on what the refined map looks like?
Where do we take the findings and the map next?
How does, and how can this process inform future changes or adaptations and who needs to know?

A key point is that this stage should be one where consensus on the refined impact pathways and casual connections should be mutually agreed. To allow this collaborative element to take shape, the research team implemented the appreciative inquiry principles of discover, define, dream and design (39). Such a process then enables the participants to make further changes to the map (if required) and then consider what happens next.

Fostering use and making Learning Actionable for the Future

The time following this workshop is crucial for those commissioning RREM to consider the findings. Instead of seeing the findings as a summative end, participants and/or stakeholders should use this stage as an opportunity to reflect, act, or plan to act on the findings to foster ‘use’ (31). As Shulha et al., (31) and Cousins (40) state, there is little point in evaluation, if the findings are not put into action. This may lead to initiative modifications, as well as identifying new areas of focus which have emerged through the preceding processes. What is more, identifying new areas and impacts of focus creates a rationale for continuing the RREM process in a new iteration, on the basis that knowledge is fallible and there should be no end to assessing the merit and worth of provision. This sparks synergy with the realist cycle (16) for developing, testing and refining theories again once an initial iteration is complete. Overall, such an approach helps to develop a continuous developmental cycle (41) of learning that runs alongside the development of a whole system approach or initiative. By the end of the RREM process, participants will have an idea of what impacts have occurred, how they have evolved over time, and what the explanation is behind them. This creates an opportunity to take ownership over this evidence and consider what this means for action and dissemination.

At the end of the RREM phase 3 the RREM community will have fulfilled the following tasks:

Prepared refined maps and respective impact pathways and connections ready for presentation in workshop 3.

Delivered workshop 3 and presented refined map(s).

Provided an environment of deliberation and consensus to agree on the maps using appreciative inquiry principles.

Produced a refined map supported by causal connections. This map may be supplemented with or embedded within an overall report summarising the three phases with the supporting data captured in phase 2 as well as stories of change.

Compiled a set of outputs that are ready for dissemination and action for future iterations of RREM and initiative design.

Having introduced the RREM protocol, in this section of the paper we consider the implications of the RREM method for practice and future direction. We will cover the following areas; practical considerations, challenges, and future directions for progressing the approach.

Practical Considerations

Staff Resource Requirements

Given that this RREM process involves the combination of two traditionally discrete methods, it is important to identify how a RREM project is appropriately resourced. This considers the role of the researchers and supporting people within the RREM team as to who is involved in what phase, and subsequent stage of RREM. Within the practical delivery of the workshops, where feasible, it is important to have a facilitator allocated to capturing information for the REM output. Specifically, there will be different orientations of this role where there will be a facilitator (or facilitators) charged with the responsibility for capturing the content that relates to the impact pathways, but also another role in beginning to shape the causal insights that represent the realist position of this work. This essentially means that facilitators need to have a skill set that is not only conducive with REM but also one that is familiar with realist evaluation. This dual facilitation will enable participants to discuss what has occurred within their work from the level of identifying impact, and then to the deeper level of understanding the how and the why. The facilitator should also guide the conversations and seek to obtain a greater level of detail. Furthermore, outside of the practical delivery of the workshops, evaluation teams need to consider who is responsible for the ‘desk based’ tasks of data analysis as well as the additional ‘realist’ orientated fieldwork data that needs to be captured. Having multi-disciplinary teams assembled who are embracing different methodological domains is fundamentally important.

Participant Recruitment

Consideration should be given to how and when participants will be recruited to take place in the RREM process. Evaluation teams will need to decide who is responsible for providing access to participant samples and data that is required for the evaluation, ideally within the discovery phase of the RREM cycle. However, this is something that needs to be managed and monitored throughout all three RREM phases. Reflection on the recruitment and engagement of participants may also be required, for example are those who are participating likely to have a bias towards the initiative (i.e., are you likely to engage participants who have had a negative experience?). It is also important to consider who the whole system approach or initiative intended to reach compared to those taking part in the evaluation.

Challenges

Although the RREM protocol presented here provides a novel method of evaluation, we must acknowledge that there are several challenges that may need to be overcome. For example, engaging with the RREM method requires the evaluator(s) to straddle the different dimensions and philosophies of both realism and pragmatism. There is the additional challenge of navigating the varied perspectives of stakeholders, commissioners and practitioners and their often different expectations for future impact and outcome from RREM. For some, RREM may simply provide a mechanism to legitimise their work (42) and showcase the positive aspects of what is being mapped. This may result in RREM being used to evidence ‘what works’ (43) as opposed to being used to explore causal connections that demonstrate the deeper causality underpinning the impact pathways, which in turn uncover negative aspects of people's work.

Second, given that the impacts can take highly variable lengths of time to emerge (44), it must be recognised that RREM has the potential to miss some longer term changes that occur. Although this can be overcome through additional RREM workshops, the majority of evaluation projects are likely to have a clearly specified end-date. Time also presents a practical challenge, in that the time available to complete an RREM evaluation will often be determined by stakeholders and therefore out of the evaluation team’s control. Evaluation teams will also be required to navigate time-expectations when they are planning and conducting their RREM evaluation.

The RREM method relies on wide active participation engagement from stakeholders, users, and facilitators. Barriers to participation may include time, cost, resources, lack of knowledge, training, and understanding of the importance of the process. Therefore, establishing a clear motivation for collaboration (31) from the outset is important for understanding why people want to be engaged and what is driving their participation. Without knowing this there is danger of stakeholders becoming what Nichols et al (45) refers to as subjugated, that is not feeling part of the process. Thus, establishing meaningful inter-professional relationships between the evaluation team and stakeholders (31, 36) is a key factor in keeping people motivated and engaged throughout the process so that there is value in the process. We should also not ignore that the RREM process also requires an evaluation team that is large enough and that is capable of completing all phases to provide a meaningful evaluation.

Future Directions

As outlined, RREM brings together principles of both REM and realist evaluation. RREM addresses the limited stakeholder accessibility that sometimes can be identified with realist evaluation, while building on the participatory and accessible nature of REM. In doing so, there is the potential for evaluators to consolidate their technical findings into plain-speaking language, which benefits not only the key stakeholders involved with the work but means that the findings can be presented to both practitioners and policy-makers, helping to bridge the gap between research and practice.

As we move forward, the next steps are to experiment with the RREM approach exploring the value it brings to the field of evaluation practice. We encourage researchers, evaluators, and wider stakeholders to apply this approach and to reflect on its use for evaluation practice. Although we outlined a model for this approach, we encourage future research to further develop RREM when applying it to individual contexts. For example, we have demonstrated that there is no time limit to how RREM should be applied and it is up to research and evaluation teams to decide what is sufficient and achievable in this respect. There is also nothing to oppose RREM approaches that perhaps only focus on discrete phases of the RREM cycle. For example, if simply implementing phase 1, it can be a valuable tool for developing causal connections and helping to design whole system approaches and initiatives in a participatory way. The same can also be said for the other phases which in their own right may offer value to a specific focus of an evaluation.

And finally, the flexibility incorporated in RREM provides an exciting and novel area for future research. We believe that an encouraging environment currently exists for exploring the utility of different complexity driven and pragmatist evaluation approaches (9) and we should seize this moment.

This paper has presented the novel merging of realist evaluation and REM to create a method which is suitable for understanding what happened, and why, with regards to public health whole of system approaches and initiatives. Through the RREM protocol, presented across the three phases, we believe that those seeking to understand the impact of their work will have a more in depth understanding of causal connections underpinning the impact pathways identified. And, they have the opportunity to do this through a highly applied, collaborative, and participatory process. We acknowledge the challenges of RREM and encourage evaluation designers to consider what is possible based on the available resources that are needed to operationalise the approach. This consideration should run alongside the level of motivation for collaboration (31) that key stakeholders have as they enter into the approach. For example, why are they participating and what is it that they wish to benefit from in a practical and or transformational evaluation sense (46)?

However, despite these challenges and practical considerations, the synthesis of realist and REM principles offers significant promise for advancing evaluation practice. Thanks to the advancement, and advocacy of complexity informed approaches to evaluation (47) there is a strong rationale for evaluation approaches such as realist evaluation being implemented in complex settings. Yet, how these complexity and theory driven approaches are mobilised in practice with wider stakeholders is open for advancement in the way that they become more accessible, meaningful and inclusive for people. And it is at this point where the pragmatic focus of REM offers value for demystifying the challenges presented by theory driven approaches such as realist evaluation. Moreover, aligning the realist approach with REM promotes this combination of hybrid methods that involves stakeholders. To reiterate a central point of this paper, the REM approach offers value for engaging people, stimulating interest which then lays the foundation for exploring causal and explanatory insight into the inner workings of programmes, policies and interventions. Nevertheless, a suitable point to end this paper echoes Barbrook-Johnson et al., (9, pg 14) advice, that : “the choice we face is to not whether to do a ‘complex’ or a ‘simple’ evaluation, but whether to do an effective or ineffective one”, and this is where RREM has the opportunity to evolve.

Ethics approval and consent to participate

Ethical approval was approved by Hartpury University and the Ethics Committee (Ref.2022-150). Informed consent for participation was gained from all participants before data was collected via Ripple Effects Mapping and interviews.

Consent for publication

Consent for publication was included in the consent forms for participants. OR Not applicable.

Availability of data and materials

All materials are included or referenced in the paper/appendix.

Competing interests

The authors declare they have no competing interests.

Funding

Hartpury University, UK, was commissioned by Active Essex, UK, to conduct a Realist Ripple Effects Mapping evaluation in April 2023. This paper presents the protocol that was utilised for the evaluation. Active Essex, the funder, supported the conceptualization and design of the protocol.

Authors' contributions

All authors (Kevin Harris, James Nobles, Louis Ryan, Christoph Szedlak, Hannah Taylor, Rowena Hawkins, Alice Cline, and Elizabeth Smith) contributed to the manuscript. All authors read and approved the final manuscript.

Acknowledgements

Authors' information (optional)

Dr Kevin Harris, Hartpury University, Hartpury, Gloucester, GL19 3BE UK.

Dr James Nobles, Health, Nutrition & Environment, Leeds Beckett University, Calverley Building, City Campus, Leeds, LS1 3HE

Louis Ryan, Sheffield Hallam University, College of Health, Wellbeing and Life Sciences

Sheffield Hallam University, 32 Collegiate Crescent, Sheffield S10 2DP

Dr Christoph Szedlak, Hartpury University, Hartpury, Gloucester GL19 3BE UK.

Hannah Taylor, Active Essex, Chelmsford, UK.

Dr Rowena Hawkins, Active Essex, Chelmsford, UK.

Dr Alice Cline, Public Health Wales, Cardiff, UK.

Dr Elizabeth Smith, Hartpury University, Hartpury, Gloucester GL19 3BE UK.

Nau T, Bauman A, Smith BJ, Bellew W. A scoping review of systems approaches for increasing physical activity in populations. Health Res Policy Syst. 2022;20(1):104.
McGill E, Er V, Penney T, Egan M, White M, Meier P, Whitehead M, Lock K, de Cuevas RA, Smith R, Savona N. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc Sci Med. 2021;272:113697.
Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, Boyd KA, Craig N, French DP, McIntosh E, Petticrew M. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374.
Gosselin V, Laberge S. Do the implementation processes of a school-based daily physical activity (DPA) program vary according to the socioeconomic context of the schools? a realist evaluation of the Active at school program. BMC Public Health. 2022;22(1):1–9.
Nobles J, Wheeler J, Dunleavy-Harris K, Holmes R, Inman-Ward A, Potts A, Hall J, Redwood S, Jago R, Foster C. Ripple effects mapping: capturing the wider impacts of systems change efforts in public health. BMC Med Res Methodol. 2022;22(1):1–4.
Rodriguez Espinosa P, King AC, Blanco-Velazquez I, Banchoff AW, Campero MI, Chen WT, Rosas LG. Engaging diverse midlife and older adults in a multilevel participatory physical activity intervention: evaluating impacts using Ripple Effects Mapping. Translational Behav Med. 2023 Apr;3:ibad018.
Washburn LT, Traywick L, Thornton L, Vincent J, Brown T. Using ripple effects mapping to evaluate a community-based health program: perspectives of program implementers. Health Promot Pract. 2020;21(4):601–10.
Willis CE, Reid S, Elliott C, Rosenberg M, Nyquist A, Jahnsen R, Girdler S. A realist evaluation of a physical activity participation intervention for children and youth with disabilities: what works, for whom, in what circumstances, and how? BMC Pediatr. 2018;18(1):1–5.
Barbrook-Johnson P, Castellani B, Hills D, Penn A, Gilbert N. Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus. Evaluation. 2021;27(1):4–17.
Chazdon S, Emery M, Hansen D, Higgins L, Sero R. A field guide to ripple effects mapping. University of Minnesota Libraries Publishing; 2017.
Hall J, Bingham DD, Seims A, Dogra SA, Burkhardt J, Nobles J, McKenna J, Bryant M, Barber SE, Daly-Smith A. A whole system approach to increasing children’s physical activity in a multi-ethnic UK city: a process evaluation protocol. BMC Public Health. 2021;21(1):1–6.
Nobles J, Fox C, Inman-Ward A, Beasley T, Redwood S, Jago R, Foster C. Navigating the river (S) of systems change: a multi-methods, qualitative evaluation exploring the implementation of a systems approach to physical activity in Gloucestershire, England. BMJ open. 2022;12(8):e063638.
Jayasinghe S, Holloway TP, Soward R, Patterson KA, Ahuja KD, Dalton L, Murray S, Hughes R, Byrne NM, Hills AP. An Ounce of Prevention is Worth a Pound of Cure: Proposal for a Social Prescribing Strategy for Obesity Prevention and Improvement in Health and Well-being. JMIR Res Protocols. 2023;12(1):e41280.
Van Belle S, Abejirinde IO, Ssennyonjo A, Srinivas PN, Hebbar P, Marchal B. How to develop a realist programme theory using Margaret Archer’s structure–agency–culture framework: The case of adolescent accountability for sexual and reproductive health in urban resource-constrained settings. Evaluation. 2023;29(3):259–75.
Hall J, Mansfield L. The benefits and complexities of integrating mixed method findings using the Pillar Integration Process: A workplace health intervention case study. J Mixed Methods Res. 2023 Aug;17:15586898231196287.
Pawson R, Tilley N. Realistic Evaluation| SAGE Publications Inc.
Phillips DC. Philosophy, science and social inquiry: Contemporary methodological controversies in social science and related applied fields of research. Pergamon; 1987.
Dalkin SM, Greenhalgh J, Jones D, Cunningham B, Lhussier M. What’s in a mechanism? Development of a key concept in realist evaluation. Implement Sci. 2015;10(1):1–7.
Greenhalgh J, Manzano A. Understanding ‘context’in realist evaluation and synthesis. Int J Soc Res Methodol. 2022;25(5):583–95.
Renmans D, Pleguezuelo VC. Methods in realist evaluation: A mapping review. Eval Program Plan. 2023;97:102209.
Harris K. Building sport for development practitioners’ capacity for undertaking monitoring and evaluation–reflections on a training programme building capacity in realist evaluation. Int J sport policy politics. 2018;10(4):795–814.
Pelletier C, Pousette A, Fox G, Keahey R, Ward K, Faulkner G, Rasali D, Allison S. Move the north: evaluation of a regional stakeholder engagement initiative to support the development of a community-partnered physical activity research agenda. Res Involv Engagem. 2019;5:1–3.
Rigby P, Van der Graaf B, Azevedo PB, Hayes L, Gardner L, Dodd-Reynolds BJ. Challenges, opportunities and solutions for local physical activity stakeholders: an implementation case study from a cross-sectoral physical activity network in Northeast England. BMC Public Health. 2020;20(1):1–4.
Punton M, Vogel I, Lloyd R. Reflections from a realist evaluation in progress: scaling ladders and stitching theory.
Harris K, Henderson S, Wink B. Mobilising Q methodology within a realist evaluation: Lessons from an empirical study. Evaluation. 2019;25(4):430–48.
Kidd L, Millar JD, Mason H, Quinn T, Gallacher KI, Jones F, Fisher RJ, Lebedis T, Barber M, Brennan K, Smith M. Protocol: Supported self-management in community stroke rehabilitation: what is it and how does it work? A protocol for a realist evaluation study. BMJ open. 2022;12(1).
Mason J, Oatley C, Harris K, Ryan L. How and why does Local Area Coordination work for people in different contexts? Methodological Innovations. 2021;14(1):2059799120985381.
Ryan L, Harris K, Henderson S. Introducing Realist Evaluation within Sport and Leisure. Evaluation in Sport and Leisure. 2022 Aug 12:115.
Dalkin SM, Jones D, Lhussier M, Cunningham B. Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol. BMJ open. 2012;2(4).
Brown S, Dalkin SM, Bate A, Bradford R, Allen C, Brittain K, Clarke A, Hand A, Protocol. Exploring and understanding the scope and value of the Parkinson’s nurse in the UK (The USP Project): a realist economic evaluation protocol. BMJ open. 2020;10(10).
Shulha LM, Whitmore E, Cousins JB, Gilbert N, al, Hudib H. Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. Am J Evaluation. 2016;37(2):193–215.
Manzano A. The craft of interviewing in realist evaluation. Evaluation. 2016;22(3):342–60.
Jagosh J. Retroductive theorizing in Pawson and Tilley's applied scientific realism. J Crit Realism. 2020;19(2):121–30.
Mukumbang FC, Marchal B, Van Belle S, van Wyk B. A realist approach to eliciting the initial programme theory of the antiretroviral treatment adherence club intervention in the Western Cape Province, South Africa. BMC Med Res Methodol. 2018;18:1–6.
Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):1–6.
Bailey A, Harris K. An autoethnographic study of realist knowledge translation within sport development. Managing Sport Leisure. 2021;26(1–2):41–59.
Schaillée H, Spaaij R, Jeanes R, Theeboom M. Knowledge translation practices, enablers, and constraints: bridging the research–practice divide in sport management. J Sport Manage. 2019;33(5):366–78.
Szedlak C, Smith MJ, Callary B. Developing a ‘letter to my younger self’to learn from the experiences of expert coaches. Qualitative Res Sport Exerc Health. 2021;13(4):569–85.
Coghlan AT, Preskill H, Tzavaras Catsambas T. An overview of appreciative inquiry in evaluation. New Dir evaluation. 2003;2003(100):5–22.
Cousins JB, Commentary. Minimizing evaluation misuse as principled practice. Am J Evaluation. 2004;25(3):391–7.
Patton MQ. Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press; 2010. Jun 14.
Jeanes R, Lindsey I. Where’s the evidence? Reflecting on monitoring and evaluation within sport-for-development. InSport, social development and peace 2014 Jul 17 (pp. 197–217). Emerald Group Publishing Limited.
Coalter F. Sports clubs, social capital and social regeneration:‘Ill-defined interventions with hard to follow outcomes’? Sport Soc. 2007;10(4):537–59.
Moore GF, Evans RE, Hawkins J, Littlecott H, Melendez-Torres GJ, Bonell C, Murphy S. From complex social interventions to interventions in complex social systems: future directions and unresolved questions for intervention development and evaluation. Evaluation. 2019;25(1):23–45.
Nicholls S, Giles AR, Sethna C. Perpetuating the ‘lack of evidence’discourse in sport for development: Privileged voices, unheard stories and subjugated knowledge. Int Rev Sociol sport. 2011;46(3):249–64.
Cousins JB, Whitmore E, Shulha L. Arguments for a common set of principles for collaborative inquiry in evaluation. Am J Evaluation. 2013;34(1):7–22.
Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, Boyd KA, Craig N, French DP, McIntosh E, Petticrew M. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374.
Bass B, Riggio R. Transformational leadership. Mahwah, NJ: Lawrence Erlbaun Associates. Inc.; 2006.
Van Dierendonck D, Stam D, Boersma P, De Windt N, Alkema J. Same difference? Exploring the differential mechanisms linking servant leadership and transformational leadership to follower outcomes. Leadersh Q. 2014;25(3):544–62.

No competing interests reported.

Appendix.docx

Download PDF

Journal Publication

published 30 Oct, 2024

Read the published version in BMC Medical Research Methodology →

Editorial decision: Revision requested
16 Aug, 2024
Reviews received at journal
15 Aug, 2024
Reviewers agreed at journal
07 Jul, 2024
Reviewers agreed at journal
01 Apr, 2024
Reviews received at journal
27 Mar, 2024
Reviewers agreed at journal
29 Feb, 2024
Reviewers invited by journal
26 Feb, 2024
Editor invited by journal
13 Feb, 2024
Editor assigned by journal
13 Feb, 2024
Submission checks completed at journal
13 Feb, 2024
First submitted to journal
09 Feb, 2024

You are reading this latest preprint version

Towards Realist-informed Ripple Effects Mapping: Positioning the approach

Status:

Journal Publication

Version 1

Abstract

Figures

Background

REM

Realist Evaluation

Bringing the ‘Real’ into REM

Methods - The RREM Protocol

Phase 1: Identifying Impacts and their Perceived Explanation

Discovery Workshops

Post RREM Workshop 1

Developing Causal Connections

Phase 2: Investigating Impacts and Their Causal Connections

RREM 2 workshop: Testing and Refining Impact Pathways and Causal Connections

Further Testing and Investigation of Impact Pathways and Causal Connections

Analysing and Synthesising Findings

Phase 3: Learning From our Impacts and Their Causes to inform practice

RREM Workshop 3: Presenting and Refining Learning

Fostering use and making Learning Actionable for the Future

Discussion

Practical Considerations

Staff Resource Requirements

Participant Recruitment

Challenges

Future Directions

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1