Overview of programme and context
The 2019/20 cycle included two in-person modules, two practical phases, and a final, semi-virtual third module, which was adapted in format and held later than planned due to COVID-19. The programme covered Myanmar, Pakistan, and Vietnam in Asia, and Burkina Faso, Cameroon, Niger, and Senegal in Africa. A total of 80 participants were divided into country teams of 10–13 people. Over 60% were men, which reflects leadership patterns in the countries concerned (e.g. for Pakistan, Niger, and Cameroon). In relation to constituencies, participants came from a variety of backgrounds, as desired, though the patterns varied by country team. The MoH and relevant social agencies (e.g. Ministries of Social Protection or Community Development) were well represented. However, the Africa region mobilised cross-cutting government representatives, such as the Ministry of Finance, the Prime Minister’s Office, the Presidency, and also parliamentarians to a greater extent, with interesting disciplinary additions such as lawyers and journalists. The private sectors (for-profit and not-for-profit) were relatively poorly represented across the board.
Contextual factors influencing programme implementation and results included, most importantly, the COVID-19 pandemic, which interrupted CAIs in some settings, distracted senior staff and caused the final module to be delayed and then to be held semi-virtually (with some country teams meeting in person but regional connections made online). COVID-19 may however also open some opportunities by raising the profile of the health sector nationally, and also normalising virtual interactions, which potentially opens new modalities of interaction for the future. Other important contextual factors noted included the ongoing insecurity in West Africa, which affected priorities in some of the participant countries (such as Burkina Faso), national elections in three participant countries, which may increase turnover of participant posts, and a growth in the P4H network, which may increase sustainability of the programme.
Relevance
We examined the extent to which L4UHC addresses priority needs at individual participant and country levels—in other words, whether it addresses a priority gap or bottleneck that is not already being met from other sources. Data to answer this question were drawn from the first set of participant interviews, as well as interviews with CFPs, core team members, and external KIs in case study countries.
Participants generally saw leadership as important to them individually and few had access to comparable training. Adaptive leadership was also highlighted as an important bottleneck for UHC at country level by global key informants (KIs), participants, and secondary sources.
‘I think there is an appetite for this, if delivered in a way that is relevant to people’s work. The work that these people are doing often comes with minimal feedback and support. It is important to our long-term development.’ (Burkina Faso KI)
‘Leadership is really an issue at all levels. You don't just need a presidential decree to make things happen. … It requires multiple actors, including civil society, but only the state can guarantee sustainability (rights and regulations). We need to find a way to dialogue and reach consensus. We have talked about UHC for 10 years but are not getting far. This needs everyone to work together.’ (global KI).
In relation to the question of whether L4UHC duplicates other programmes, a few candidate programmes were identified but all have distinctive approaches. There was thus a consensus that L4UHC offers something that potentially adds value and is complementary to other efforts.
Effectiveness
We examined the extent to which L4UHC has met its objectives at output and outcome level, as well as how these were achieved, for whom, and what factors were important (positively or negatively), both from the programme side and externally. This section draws on all main data sources—participants and wider KIs, the survey, case studies, and secondary sources.
Outputs
Our first section focuses on the individual level, examining the evidence for changes in commitment, competence, and understanding by programme participants, followed by a discussion of explanatory factors relating to the way the programme was run and feedback on the various components. This section draws on all evaluation data sources, but especially on survey data and interviews.
Individual commitment to UHC was already reported as high for most participants at the start of the programme—in that sense, selection was good, and some had long careers supporting UHC—which meant there was less scope for gain respecting this dimension. Change in commitment for some individuals was noted, however, along with growth in understanding.
Individual competences were considered in relation to leadership, such as ability and confidence in areas including stakeholder mapping, listening and dialogue, coalition building, and change management. Open answers from the three module surveys highlighted listening skills as one of the main competences participants felt they had gained across both regions, with coalition building increasing by Module 3 (Table 4).
Table 4: Top skill gained (Module 3 participant presentations)
Skills
|
Burkina
|
Cameroon
|
Niger
|
Senegal
|
Myanmar
|
Pakistan
|
Total
|
Deep listening
|
2
|
3
|
6
|
3
|
6
|
6
|
26
|
Managing adaptive challenges
|
1
|
3
|
2
|
|
3
|
3
|
12
|
Creating coalitions
|
|
2
|
2
|
3
|
|
5
|
12
|
Stakeholder mapping
|
4
|
|
|
1
|
2
|
2
|
9
|
Self-management
|
1
|
1
|
|
|
3
|
1
|
6
|
Total respondents
|
8
|
9
|
10
|
7
|
7
|
10
|
51
|
Analysis of participant interviews at the start and end of the programme indicated that, in many cases, participants had considerable skills in leadership relating to their ongoing roles at the start, but all had been able to develop or deepen these skills in some way. Unsurprisingly, some of those who were less experienced at the start reported learning most. Feedback from those working with the groups also suggested improvements in listening and dialogue, including increase respect for minority positions and greater involvement of those with lesser perceived status at the start of the programme.
Regarding the understanding of UHC, slightly different patterns emerged from Asia and Africa, which may be explained by the composition of the groups. There was more evidence of a development of understanding in the Africa group as a whole (though there are individual examples from Asia). This may be because the participants in Africa came from more diverse organisational backgrounds, which meant some had a more limited starting understanding of UHC.
Most participants in Asia started with a good basic understanding of UHC, although it was closely equated with health insurance in some settings, which can limit a broader understanding of the concept, the different routes that it can take, and its complexity. For some individuals with lower starting levels, we identified growth in comprehension, but this was not clear across the board.
Explanatory factors
We considered how the output changes noted above were achieved, what worked well (or not), and what lessons emerged. We examine evidence relating to the assumptions in our theory of change at this level—for example, how well components were delivered and how well participants were selected—as well as the role of external factors, such as country contexts. Although these are discussed under outputs, given the connections across the theory of change, they also influence higher levels in the theory of change, such as outcomes and impact. Some key findings are highlighted here.
Overall design: This was seen by participants and facilitators as good, but with some tensions perceived between a focus on individual personal development versus a focus on teams achieving concrete results in their action plans, and also (especially initially) on the balance of adaptive and technical content. Some participants were also not clear on the approach of L4UHC at the start of the programme but appreciation grew with understanding.
‘I really like the three modules and the design, starting with not heavy topics, [which] makes participants increase their interest and motivation, and later try to integrate with more technical aspects.’ (Myanmar participant)
Country selection: In general, informants felt the countries in Africa had been appropriately selected—all faced different challenges, but there was room in each for benefits from participation. Less feedback was received on the Asia country selection.
Participant selection: L4UHC aims to target senior individuals in a range of UHC-related institutions, who have time to attend and participate consistently. Overall feedback on the make-up of teams was positive, although some gaps were noted in each case by participants.
‘They chose participants well—hard-working, activists, well-motivated to improve life in Niger, also representing all sectors.’ (Niger KI)
Gaps related to a variety of factors, including (from the programme side) the institutional links that L4UHC partners in-country have and how they mobilise them (or not), as well as overall limits on numbers of participants and financing constraints. From the country side, the need for language skills, time availability of senior staff, and internal hierarchies and procedures were influential.
Continuity of participation: Continuity of participation in modules was good in general. For CAIs, it was more partial, with many teams led by an active core of a few participants.
Host countries: Views on these were largely positive, with some reservations expressed on the sites in Asia.
Preparation pre-programme: Thoroughness in preparing candidates for the start of the programme varied across the countries (from very good to very limited).
Modules overall: Overall, participant satisfaction with the modules was high—close to 60% rated the modules as excellent or good overall, with satisfaction highest for Module 2 and lowest for Module 3 (which can be understood in relation to the disappointment regarding this module moving from in-person to semi-virtual due to COVID-19). Considerable variation was shown across countries, from 25% in Vietnam to 93% in Pakistan (Table 5).
Table 5: Overall rating of modules (% rating them excellent or good), by country
|
Burkina Faso
|
Cameroon
|
Niger
|
Senegal
|
Myanmar
|
Pakistan
|
Vietnam
|
Average
|
M1
|
88.9
|
41.7
|
88.9
|
57.1
|
63.6
|
88.9
|
23.1
|
64.6
|
M2
|
60
|
77.8
|
100
|
33.3
|
70
|
100
|
55.6
|
71.0
|
M3
|
14.3
|
50
|
83.3
|
42.9
|
22.2
|
88.9
|
No data
|
43.1
|
Average
|
54.4
|
56.5
|
90.7
|
44.4
|
51.9
|
92.6
|
26.2
|
59.5
|
Note: Scores represent the proportion rating the module overall as excellent or good in the end-of-module surveys. The Vietnam team did not attend Module 3, hence the data gap. Averages have been adjusted for this.
It is also interesting to examine these scores by the profile and sector of participants. Senior management had the highest overall rating (56% rating the modules as excellent or good), compared to 51% for operational staff. Across the sectors, the private sector gave the highest ratings of excellent or good (71%) and civil society had the lowest (46%), although we need to note low numbers in both these categories. Another overall satisfaction metric was the proportion of participants who would recommend the programme to others. The average here was high, at just under 85%.
Module content: Participants were asked about each component in the modules, on four domains—whether the exercise was engaging, whether its delivery was well-paced, whether the content was relevant to them, and whether they had learned something substantive they felt they could apply. Aggregate scores indicated high overall satisfaction, but more so in Africa than Asia. Surveys and interviews suggest that participants most enjoyed the participative and experiential components, as well as those which allowed for sharing across settings and teams.
After some debate about adapting the format to COVID-19 restrictions, the regions met virtually for module 3, though with some country teams meeting in person in one place, where this was possible. This allowed for some intimacy, although of course disappointment was expressed by participants because the full modules were not held. Survey responses, aggregating across all components in the module, were positive across both regions, though there was a perception by a significant minority of lack of interactivity. One country delegation did not choose to join this final module.
Organisation/facilitation: Overall, modules were seen as well-facilitated and structured, although responses varied by module and country, and there were some dissenting voices for Module 3 in particular. A large proportion were satisfied with the practical arrangements, but there was a variety of patterns across countries and modules, with more dissenting voices in Africa (relating to per diems and travel).
Practical phases: Between the modules, participants were encouraged to undertake activities as a group (including stakeholder mapping, sensing journeys, and other consultations) to develop preliminary ideas for their collective action initiatives (CAIs) in phase 1 and then to implement them in phase 2. When questioned about these activities in the survey, most people reported undertaking them and finding them useful, a lower proportion felt they had had sufficient support from their country focal persons, particularly in two countries.
CFPs and coaches: In general, the role of CFPs appeared to be more central in the Africa cycle. Clear roles were played in relation to each country and there was consistency and reasonable intensity in the support provided by the nominated CFPs (who combined this role with being the local P4H focal points). Perhaps partly as a result, the role of coaches was weaker in this region, which was not the case in the Asia cycle. In Myanmar, the roles of the P4H focal point, the L4UHC CFP, and the local coach were assigned to one person. In some settings, such as Pakistan, there was a perceived overlap between the role of the CFP and the role of the coach. In Asia, the regional coach helped provide momentum and support country-based coaches.
External factors: The COVID-19 pandemic was a major factor, reducing personal contact in the second practical phase and in Module 3, while also absorbing participants with more urgent tasks.
‘The major challenge is the attention span, which has reduced dramatically with COVID-19. Also, online digital meetings are the opposite of building a real trust and deep collaborative spirit within a team. This presents a real challenge for L4UHC and [for] building personal relationships.’ (global KI)
Country buy-in was also important but was notably absent in Vietnam. Cultural assumptions about leadership, P4H partners’ alignment in participant countries, and language barriers were all noted as important external factors influencing the programme results.
Intermediate outcomes
For these, we drew mainly on KI interviews with participants and other stakeholders.
Team development: Global KIs perceived a substantial team-building effect for participants in several countries, although this was varied across teams and regions.
‘They often arrive at the table with a lot of distrust across organisations. They then gradually move from talking from their organisational perspective to talking about the team.’ (global KI)
This is a domain where we observed considerable change, especially in relation to the teams in Africa, where strong bonds within the group appear to have been formed in some of the teams, despite (or maybe because of) their diverse institutional backgrounds.
‘The programme helped create a team, it didn’t however completely manage the conflicts but [it] did make them more explicit. The whole team could see them and discuss them. So not a perfect dialogue, but the dynamic helped to manage them within the “family”. The dynamic was good—everyone was active and applied. … A real sense of being a family!’ (global KI)
The teams in Asia presented a more complex picture in that the Vietnam team did not appear to have worked together in a coordinated way except through other national fora. In Pakistan, strong teamwork emerged, but centred around two CAIs more than at whole-team level. In Myanmar, collaboration was emerging but perhaps at early stages, also given the interruptions of COVID-19. The course structure and the diversity of teams were seen as enablers to change in this domain, while the constraints of hierarchy were barriers to equitable group work, especially in some settings, like Pakistan. Some increase in team members’ wider networks was also reported across all countries.
National coalitions: More ambitious than team building, or building links between team members, was the question of whether L4UHC has contributed to a wider shift in national coalitions working for UHC. This could have been through engaging other actors in their meetings, their CAIs, or other activities. In general, this was less reported on by stakeholders, so the findings here are limited, and it may be that this is too ambitious an expectation for this stage in the programme. Limited change in national coalitions was found in general, but in some settings, such as Burkina Faso, the group may be helping to reduce tensions between institutional over mandates, as well as increasing the capacity of individuals within existing national networks.
‘There was already a multisectoral group working on UHC before, so that hasn’t changed much, but L4UHC has reinforced the capacities of those in the group who were in that multisectoral forum.’ (Cameroon KI)
Regional networks: A second aspiration within the theory of change was for the L4UHC programme to strengthen regional UHC networks, which could occur via strengthening the links of individuals or teams to wider networks, or by creating a network across L4UHC country teams. There was weak evidence for change in this respect in this cycle. National grouping appeared to be prioritised in group work in modules, and there was no encouragement specifically to communicate across countries between modules. Some personal relationships were established across teams, but these were limited. KIs agreed this could be given more explicit priority in future.
Higher-level outcomes
We examined the impact of L4UHC on policy and practice in participating countries, particularly though the lens of the CAIs. Overall, there was good engagement in significant national processes by the Africa teams, but limited concrete results where a clear contribution of the L4UHC Team could be traced as yet, with the exception of Burkina Faso, where the group appeared to have facilitated important progress on Universal Health Insurance development and the response to COVID-19. The Burkina Faso team chose to focus their CAI on improving the operation of the policy to cover indigents in the Universal Health Coverage programme, including supporting the creation of a shared platform to coordinate actors and ensure full coverage of the country. In June 2020, the group reported that contracts had been signed with implementing organisations and that the L4UHC Team had supported awareness raising around the new policy. Results reported in October 2020 included the launch of the policy to cover indigents (in September 2020).
In Pakistan, the team was able to achieve concrete gains in two areas – increasing enrolment into the national health insurance programme in Rawalpindi district by novel and networked approaches to enrolment (growing from 40% to 59% coverage over summer of 2020), and setting up a draft memorandum of understanding between the State Life Insurance Company (the main insurer working on UHC) and Indus Health Network, a major private not-for-profit hospital network, to increase the network of UHC providers - while the CAIs were still ongoing with few results to show as yet in Myanmar, and did not progress in Vietnam.
Factors favouring success included having clearly defined CAIs, having CAIs with achievements linked to capacities held by group members, working with the grain of political priorities, and having a strong team dynamic drawing from diverse group skills and networks. Constraining factors included the impact of COVID-19, lack of resources to support team activities, and unsupportive institutional contexts.
Efficiency
We examined the overall costs of L4UHC, make judgements (insofar as it is possible to do so) on its value for money, and identify some potential areas for efficiency gains. Evidence for this came from the L4UHC financial reporting and also from KI interviews with the management team.
The two core funders of L4UHC were SDC and GIZ. They co-financed the core costs of L4UHC, including the cost of the full-time global coordinator, the events costs (running of the modules), the costs of the coaches, and the monitoring, evaluation, and communications. Relatively smaller contributions came from WHO, for the time of core team members, from the World Bank, for the time of the didactic lead and also for some country delegations (Niger in this round), and from USAID and Expertise France, mainly covering the time of their staff, who provide time inputs to support the programme.
The overall cost of the programme for this cycle was just over €2.1 million for the full programme, which equated to €26,705 per participant. This cost included all the staff and travel time provided by partner organisations to support the programme (commonly omitted from costings). The largest element was the event costs—the running of the modules—which absorbed around 43% of overall resources.
The programme was relatively expensive compared to short training programmes but the L4UHC programme takes a longer and more ambitious approach, and there are few good comparators/benchmarks by which to judge its value for money. Making a judgement on value for money is challenging for a programme like L4UHC, which has no direct comparators and with outputs and outcomes that are multiple, potentially long-term, and in some part intangible. Potential gains over time through reforms to improve health, health access, social equity, and saved waste in the health sector of participating countries could easily repay investments many times over. Equally, these gains are not guaranteed and longer-term outcomes are unpredictable.
As important as judging value for money may be identifying ways of increasing its efficiency, for example through establishing it as an independent entity to reduce overhead costs, streamlining decision making, and clarifying roles and responsibilities within it.
Impact
We examined the likely longer-term impact of L4UHC and any unintended effects that have been observed (positive or negative). The evidence here drew on KIs across different constituencies.
In Asia, there was limited optimism about impact in Vietnam. In Myanmar, there was judged to be a possibility of improved longer-term collaboration across actors on UHC, although quite strong central control over key decisions make this challenging in the short run (and the current military coup in 2021 has certainly not helped). In Pakistan, there was strong momentum for extension of health insurance, which is likely to continue, with L4UHC having contributed to strengthened personal networks for some actors to support this movement.
In Africa, some KIs saw a shift in attention to those in most need, to which L4UHC may have contributed. It is plausible that the cross-sectoral teams established in all four settings may play catalytic roles in future policy development, funding, and implementation. Consideration of how to maintain the team work established to date in L4UHC is now required.
Risks to impact come from changing posts and staff turnover for participants, but also from shifting political agendas and lack of policy continuity.
KIs were asked if they had observed any negative unintended effects, but none was raised. On the positive side, however, a number of unintended effects have been noted:
- A resource for responding to COVID-19—for example, in Burkina, one KI pointed out that, as the group was already in existence, it was able to support better communication and more harmonised messaging with the population across different actors.
- In relation to their role as P4H focal points, two CFPs stated that their work on the programme had extended their networks and contacts, with side benefits for their work for P4H.
- Personal development for resource people—although the programme understandably focuses on participants, some of those helping to organise and run it also reported some personal benefits in terms of changed perspective and heightened skills (e.g. in facilitation).
- Career development—helping participants progress their career was not a direct aim, but it was pleasing to hear at least one concrete example of a participant who had been promoted and who attributed this to a growth in skills and confidence supported by the programme.
- Side benefits to older participating cohorts—members of the Nepal and Chad teams who were alumni of the programme joined some activities in 2020, which aimed to provide inspiration for the current cohort but also to stimulate continued actions by those teams. We did not examine the extent to which this was effective, but note this as a potential positive effect.
In addition, we found some evidence of benefits from participation for the participants’ organisations. This was not a central expectation in the theory of change because of the individual and cross-organisational design of L4UHC. The main mechanisms for transmitting benefits were: the person being more effective in their core work role (skills, confidence, networks), sharing learning (substantive, also on L4UHC methods) from the programme with colleagues, and strengthening their engagement in other activities (e.g. voluntary and domestic).
‘After the course, my boss acknowledged my role and involved me more in decisions. This did not use to be the case, but I am now involved in policymaking. ….’ (Pakistan KI)
Sustainability
Finally, we considered how likely it is that the benefits noted from L4UHC would continue after the end of the programme—which reflects commitment by various parties, capacity, and also affordability—and how sustainability could be enhanced. This section draws on KI interviews, and also on participant presentations in Module 3.
Overall sustainability varied by level. In relation to individual commitment, this is likely to remain high, with benefits for the country, depending on future postings. The team dynamic appears to be more secure in Burkina Faso than in other settings, where some concerns around staff turnover, resourcing and institutional base were expressed.
‘It would be good to keep the group connected, also find a localised way to expand the conversation, to making it indigenous beyond the trained group, who will be dependent on their jobs. Group conversations are very limited to workshops and meetings—there is no thinking within the group on how to take it forward.’ (Pakistan KI)
All teams laid out plans for follow up actions. Some represented a completion of CAIs planned during the programme (for example, for Myanmar, Niger, Cameroon, and Senegal). Others took ambitions further and were more concrete (e.g. for Pakistan and Burkina Faso). Based on the findings in this evaluation, it seems likely that momentum will be sustained in the countries with more concrete plans, which is also indicative of more specific understanding of blockages and areas with potential for movement.
In relation to overall funding, development partners were supportive, although L4UHC has been heavily dependent financially on two partners and would benefit from diversifying its funding sources and potentially establishing itself as an independent entity, as well as increasing awareness in partners and potential partners of its work.