Involving the public in research prioritisation and as research partners in the Wales COVID-19 Evidence Centre (2021-23): processes, research priorities, outputs, and lessons

doi:10.21203/rs.3.rs-3624819/v1

Download PDF

Research Article

Involving the public in research prioritisation and as research partners in the Wales COVID-19 Evidence Centre (2021-23): processes, research priorities, outputs, and lessons

https://doi.org/10.21203/rs.3.rs-3624819/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

High quality health and care research must involve patients and the public, but the COVID-19 pandemic presented challenges to usual involvement methods. We needed to identify new yet inclusive ways of involving the public in every stage of the Wales COVID-19 Evidence Centre’s (WCEC) rapid evidence generation processes, whilst fulfilling the UK Standards for Public Involvement. We describe how the public was involved in research prioritisation and as research partners in rapid research processes, and present key learning and recommendations.

Methods

Methods differed for two broad categories of WCEC public involvement opportunities. We recruited the public to our Public Partnership Group (PPG) to support activities throughout the research-knowledge mobilisation process. We recruited people from underserved groups impacted most by the pandemic to our Public Consultation Exercise; they participated in virtual or face-to-face workshops to identify and rank priority evidence needs. Involvement activities were mapped to the six UK Standards for Public Involvement. Learning was captured via quarterly meetings with our public partners, progress reports, and our research Collaborating Partners.

Results

We successfully involved 99 members of the public in identifying COVID-19 research priorities and eight members of the public to our Public Partnership Group, who supported our rapid evidence activities across the research-knowledge mobilisation process. In a pandemic context, we implemented involvement processes that were inclusive and accessible, utilised virtual platforms and methods, supported the development of new skills for our public partners, and fulfilled the UK Standards for Public Involvement. Our findings highlight the essential and complementary input of the public in identifying research priorities. Key challenges included accommodating urgent research requests from policy / practice leads, balancing views of all stakeholders, identifying answerable research questions, and capturing impact.

Conclusion

We provide a blueprint for involving the public across the research-knowledge mobilisation process in rapid research prioritisation and evidence environments. Key challenges that require further action include tailored guidance and training for involving members of the public in rapid evidence environments, considering the impact of rapid involvement opportunities on research inclusivity and diversity of the participants, and how we can capture impact of public involvement in contexts where rapid knowledge mobilisation is paramount. , whilst minimising burden on public partners.

Coproduction

COVID-19

lay involvement

priority setting

priority setting exercise

public engagement

public involvement

PPI

PPIE

public partners

research prioritisation

knowledge mobilisation

UK Standards for Public Involvement

During the COVID-19 pandemic, the Wales COVID-19 Evidence Centre (WCEC) needed to produce evidence quicker than ever before to help policy makers and health and care leaders to make decisions. High quality health and care research must involve patients and the public, but the speed of our research meant we could not involve them in the usual ways. In this paper we describe a) how a core Public Partnership Group supported every stage of our rapid evidence processes and b) how we worked with underserved groups most impacted by the pandemic to identify and rank the most important evidence needs – what mattered most to these members of the public?

We successfully involved a total of 99 members of the public in identifying COVID-19 research priorities and eight members of the public in our Public Partnership Group, who worked directly with research teams from start to finish. We offered involvement opportunities that were inclusive and accessible, that used virtual methods to communicate, supported new skills development for our public partners, and met the six UK Standards for Public Involvement. Our work also highlights the complementary role the public play in identifying research priorities. Key challenges included accommodating urgent research requests, balancing views of all stakeholders, identifying specific research questions, and capturing the impact the public involvement made. We provide a blueprint for involving the public in rapid research prioritisation and evidence environments, but more work is needed to consider the impact of rapid involvement opportunities on inclusivity and diversity.

The COVID-19 pandemic highlighted the importance of health and care research in a way that we have not seen previously. Policy makers and health and social care leaders needed to make rapid decisions based on the best available evidence. High quality health and care research must involve patients and the public.¹The move to virtual communication, evolving health and care evidence needs, and the rapid decision-making processes of the pandemic presented challenges to involving members of the public in the usual ways, whilst also offering new opportunities for involvement. If the public were to remain key partners in our research, we needed to identify new yet inclusive ways of involving them in every stage of our rapid evidence generation processes, from research prioritisation through to knowledge mobilisation, whilst fulfilling the UK Standards for Public Involvement.²

Evidence was needed rapidly to inform decisions about managing the COVID-19 pandemic and its impact, especially during the earlier, high threat and uncertain periods. However, evidence was not always readily available to decision makers, and the exponential increase in COVID-19-related research outputs every month made it challenging. Funded by Welsh Government, the Wales COVID-19 Evidence Centre (WCEC)³ was established in March 2021 with the remit of rapidly synthesising and generating relevant and high quality COVID-19 evidence to answer the most important policy and practice questions for Wales. Our work consisted of two main areas: evidence synthesis, where we rapidly brought together evidence from studies that were already published (typically three months), and rapid primary research, where we conducted small-scale new studies to answer important COVID-19 questions for Wales.

Patients and the public are critical partners for high quality and relevant research.¹ For this report and WCEC processes, we define public involvement as a ‘process that empowers members of the public with relevant experience or knowledge to contribute directly to how the research is prioritised, designed, conducted, analysed, and disseminated, leading to a real partnership between researcher and public contributor, and shared responsibilities in the decision-making process’.⁴ Public involvement meeting this definition ensures high quality research that answers real world and relevant questions for the public. We need to know we are asking the right questions in the right way and at the right time, that we can accurately elicit and capture potential benefits to the public, and that we are sharing our outputs in accessible and understandable ways. Unlike other health and social care research topics, where a subgroup of the population has an interest in or lived experience of an issue, everyone experienced and was impacted in some way by COVID-19, whether that be physical health, social, mental health, economic, educational, or other impacts. We could only learn to live with and recover from COVID-19 by learning from those who lived through the pandemic, and peoples lived experiences were critical in guiding and overseeing our work programme.

National standards have been developed to improve the quality and consistency of public involvement in research,² various good practice guidelines and toolkits exist, ^{(e.g. 2,5,6)} and as a research team we also had significant combined experience of working with members of the public as research partners and leadership roles in this area.^{(e.g. 7,8)} Typically, an evidence synthesis research project (review of existing evidence) takes around 12 months, and primary research (studies that collect new data) may take even longer. Well-established research prioritisation exercises that involve the public also exist (e.g. James Lind Alliance Priority-Setting Partnership process),⁹ usually taking between 12-18 months to conduct.

However, the pandemic was rapidly changing, and evidence priorities were quickly changing, evolving from evidence gaps about how we control the virus, to the impact of these measures, to how we recover health and care services from the pandemic impact within a 12-month period. Further, pandemic decision makers required a rapid turnaround of evidence for it to be relevant and useful. Instead of years, we were now being asked to provide evidence within a few months, sometimes even weeks for urgent evidence needs.

Alongside the rapid timeframe was the move to remote communication. Due to pandemic restrictions, we were unable to meet face-to-face, and so all our public involvement opportunities moved online. This typically involved video meetings via Microsoft Teams or Zoom and correspondence via email or online file hosting services (e.g. Microsoft Teams). Prior to the pandemic, virtual involvement opportunities for public partners were more limited, and there was little guidance on how to support partners effectively via online platforms. We needed to make sure that we were supporting our public partners to take part, whilst also providing inclusive opportunities that promoted diversity and met the UK standards.²

Despite our remit of rapidly delivering research for policy and practice, and the short time frame between identifying an evidence need, conducting the research, and sharing the findings with decision makers, public involvement remained critical to WCEC processes. We wanted to ensure inclusive involvement in all our processes, from start to finish, that fulfilled UK Standards for Public Involvement.² To achieve this, we needed to recruit a core group of public partners who were readily available and supported to inform our processes, but also to consult the broader public on their research priorities; particularly those groups that had been most impacted by the pandemic, though often also not included in research e.g. disabled people and people from ethnic minority communities.^10,11

In this paper, we describe how the public was involved in research prioritisation and as research partners in the rapid WCEC research processes. Our objectives were to describe:

how the public was involved in COVID-19 research prioritisation
how public members were involved as research partners in our rapid COVID-19 research and knowledge translation activities
the outputs of these activities and how they informed the WCEC work programme and COVID-19 policy and practice decision making
how our activities met UK Standards for Public Involvement
lessons learnt from involving the public in COVID-19 research prioritisation and in rapid research and knowledge mobilisation activities

Methods differed for the two broad categories of WCEC public involvement opportunities: the Public Partnership Group (PPG) and the Public Consultation Exercise. These are described separately below. Ethical approval was not required for the involvement activities outlined.

Public Partnership Group

The PPG opportunity was advertised to registered members of the Health and Care Research Wales Involvement Community via email and their online involvement opportunity database. This included a summary of the project (the Evidence Centre), role expectations, expected commitment, and support provided. This paid opportunity was categorised as ‘green’, requiring applicants to have some experience and understanding of public involvement, prior basic involvement training, commitment to confidentiality, understanding of health and care research delivery, and the ability to attend online meetings.⁶ Interested members applied, and anonymised application forms were passed to the WCEC team. We aimed to recruit a group of between eight and 12 members.

Following assessment by the WCEC core team against the criteria listed, applicants were invited to join the WCEC PPG. Members were given a Terms of Reference, agreed to a ‘Code of Conduct’, and attended a welcome / induction meeting. They were asked to complete a ‘Working Together’ needs assessment, which allowed us to understand support required to work with us, communication preferences, accessibility requirements, and topic preferences (accounting for the sensitive nature of our research areas e.g. bereavement during the COVID-19 pandemic).

PPG members were involved in all our processes throughout the research - knowledge mobilisation cycle (see Figure 1). Key PPG tasks included research prioritisation, contributing as research partners in evidence synthesis / primary research, co-writing reports / peer-reviewed publications, writing lay summaries, reviewing infographics, presenting at regular WCEC symposia, panel members at Welsh Government webinar evidence briefings, WCEC governance, and co-delivering at public engagement events (see Table 1 for further details about each activity). As noted, the WCEC work was rapid and reactive; we needed to address urgent pandemic evidence needs as they arose. Typically, the turnaround time for an evidence product was two-three months, but, for urgent policy evidence needs (e.g. evidence on vaccination for pregnant women),¹² this could be as little as two weeks. PPG members were informed about new or upcoming opportunities via email, clearly outlining timeframe and commitment required. We adopted a fair and flexible approach to share the workload. Work allocation always considered members’ existing work programme commitments, other personal commitments, and their preferences noted in the ‘Working Together’ needs assessment. In addition to regular work programme meetings (when a member is allocated to a project), quarterly PPG catch-up meetings were arranged to keep in touch, discuss ongoing work, and reflect on and receive feedback on suggested improvements to our processes.

<<INSERT FIGURE 1>>

Figure 1: PPG members involvement throughout the research-knowledge mobilisation cycle

Public Consultation Exercise

Public views on COVID-19 research priorities were sought through our Stakeholder Priorities Consultation Exercise and targeted workshops with groups most impacted by the pandemic.

Stakeholder Priorities Consultation Exercise

We initiated a Stakeholder Priorities Consultation Exercise (May 2021) to identify COVID-19 evidence needs across key stakeholders, including policy makers and advisors, health and social care leads, third sector organisations, and the public. We first consulted the public at the Health and Care Research Wales Public Forum (May 2021). A survey was sent to registered attendees ahead of the event asking them to list COVID-19 research priorities, with accompanying guidance. Survey feedback was collated, and participants were asked to complete an online ranking exercise during the event to identify their most important priorities. Further, we consulted the Wales Centre for Primary and Emergency Care Research public involvement group (Service Users for Primary and Emergency Care Research; ‘SUPER’ Group) between May – October 2021. This group was also asked to submit a standard proforma used by all stakeholders, describing up to 10 top research priorities and information about why the research was important and who might benefit.

Proposed priorities were returned to the WCEC, allocated a theme, and were considered for inclusion in the work programme alongside priorities submitted by policy, health and social care stakeholders against five standard criteria: 1) the relevance to the current or predicted future COVID-19 context; 2) the importance of the evidence gap; 3) the potential benefits of the question being answered (potential impact); 4) how the evidence would be used (knowledge mobilisation plans); and 5) the timeframe required for the evidence to be useful. Two members of the PPG were involved in this assessment and decision-making process (see Table 1). Our agreed work programme was made publicly available on our website.³

Workshops with groups most impacted by the pandemic

Certain groups and communities were disproportionately affected by the pandemic,^10,11 and so we targeted the following groups for recruitment to one of four dedicated prioritisation workshops (Spring 2022): 1) children and young people (representatives from Children’s Commissioner for Wales, the Wolfson Centre for Young People’s Mental Health, and the Advice Leading to Public Health Advancement, or ALPHA, a young people advisory group); 2) Taff Housing (affordable housing and support provider in South Wales); 3) the Ethnic Minorities and Youth Support Team (EYST) Wales; 4) and Disability Wales (national association of disabled peoples’ organisations in Wales). Using existing links with these groups, email invitations were sent via Health and Care Research Wales’ Senior Public Involvement Manager (PG) to participate in a workshop (separate workshop for each group).

Workshops took place face-to-face or online and across one or two sessions, depending on the group’s preference (indoor group face-to-face contact was again possible when these workshops were conducted; Spring 2022). To enable participation, we provided sign language and palantypist support (speech-to-text) for the Disability Wales workshop. Prior to the workshops (face-to-face and online), participants were sent a summary of what the workshop(s) would involve, an optional and anonymised equality monitoring form, and we asked permission to use the workshop discussions to inform the WCEC work programme. The following questions were asked during the workshop(s), via facilitated small and larger group discussions: 1) what were your experiences of the pandemic? 2) which of these issues matter most to you, your family / friends, and community? and 3) what were the most important COVID-19 research priorities? Online workshops were recorded (with permission); facilitators captured discussions during face-to-face and online workshops to identify discussion themes and inform the next stage.

Following each workshop(s), feedback was collated into a list of research priorities, and participants from each workshop were asked to rank their top 10 priorities using an online form (paper version was also available for face-to-face discussion). The top ten research priorities agreed by each group were fed-back and were considered for inclusion in the WCEC work programme (see previous section for criteria). Groups were informed about questions they had proposed that had been included in the WCEC work programme (including how they will be answered / timeframe), questions already answered elsewhere (e.g. other published or ongoing research), and any impact from the work (process currently ongoing); see Figure 2 for an example of how the information was shared with one of our groups.

Question refinement

Priorities proposed for our work programme were often refined in line with research guidelines for formatting researchable questions,¹³ to ensure they were answerable, specific (for example, making sure that the population and outcomes were clearly defined), and had a clear pathway to impact (i.e. the evidence could be used by policy and practice leads). Occasionally, multiple stakeholders submitted similar research priorities; in this instance, research priorities were merged.

<<INSERT FIGURE 2 HERE>>

Figure 2: Format of the feedback provided to workshop groups on questions proposed that have been included in the WCEC work programme – example from the Disability Wales workshop

Meeting UK Standards for Public Involvement

We mapped each involvement activity (across the PPG and public consultation exercise) to the six UK Standards for Public Involvement² and described whether and how we met them. The six standards are: 1) Inclusive Opportunities; 2) Working Together; 3) Support and Learning; 4) Governance; 5) Communications; 6) Impact.

Capturing learning – what works and does not work well?

To understand what was working well and areas that required improvement across our public involvement activities, we held quarterly online PPG meetings (via Teams). Members were encouraged to critically reflect on their experiences and on our processes, including key challenges and possible solutions. Meetings were recorded and detailed minutes were captured. Following every meeting, minutes were shared with the PPG, along with a “You said, we did” summary document, highlighting key changes made or further actions to be taken. PPG members were also encouraged to provide ad-hoc feedback during study meetings or email, which would be fed back to the Evidence Centre management team at their weekly meetings. PPG members also contributed to our regular progress reports, allowing them to comment on what had gone well and any recommended challenges, and views were sought from our Collaborating Partner groups at regular meetings. Regular funder progress reports were collected to identify key learning specific to public involvement.

Public Partnership Group

We recruited 10 members to the PPG; two members later withdrew due to other commitments. Details of the type and volume of WCEC research and knowledge translation activities supported by our PPG members are included in Table 1. In summary, the PPG supported 24 evidence reviews and three primary research studies, wrote 34 lay summaries, co-wrote 17 final reports, co-authored 28 publications (peer-reviewed articles and pre-prints), attended six Welsh Government Evidence Briefings, and contributed to three Public Symposia. Twenty-two of the reports published by the WCEC³ have been cited in Welsh Government publications, demonstrating knowledge mobilisation and impact.

<<INSERT TABLE 1 HERE>>

Table 1: A summary of WCEC PPG activities and outputs throughout the research-knowledge mobilisation process

Research Prioritisation Exercise

Stakeholder Priorities Consultation Exercise

At the Health and Care Research Wales Public Forum (May 2021), 15 people contributed to the survey to identify priorities and 33 participated in the online ranking exercise. Of the ten priorities proposed by this exercise (see Supplementary Table 1), four were included in the WCEC work programme between May 2021 and March 2023. Of the five priorities proposed by the SUPER Group (see Supplementary Table 1) in October 2021, one was included in our work programme (see Table 2). Rationale for inclusion / exclusion can be found in Supplementary Table 1. All five research priorities posed by the public during the Stakeholder Priorities Consultation Exercise and included in our work programme are now complete with outputs (see Table 2 for links).

Workshops with groups impacted most by the pandemic

Workshops (total of seven across four groups) were conducted with children and young people representatives (n=18), Taff Housing (n=8), EYST (N=13), and Disability Wales (n=15) between March – May 2022, with a total of 54 attendees contributing across all groups (see supplementary Table 1 for full workshop details and the top 10 ranked research priorities proposed by each group). Of the total 40 research priorities proposed by the four workshop groups, eight were included in the WCEC work programme (see Table 2). Priorities were included for each of the four groups. To date, seven are complete with outputs and one is nearing completion (see Table 2).

Public priority themes

Bringing together the results of the Stakeholder Priorities Consultation Exercise (wider public) and the workshops (public groups impacted most by the pandemic), a total of 99 public members proposed 55 research priorities covering 11 themes (Supplementary Table 1). The top five themes proposed included: health and care service delivery (n=13), social impact (n=9), health impact (n=6), mental health and wellbeing impact (n=6), and inequalities (n=5). Priorities related to health and care service delivery and inequalities were proposed by five of the six public groups and priorities related to the impact on health or mental health and wellbeing were proposed by four groups. Priorities related to social impact were proposed by three groups, but were notably important for the Taff Housing group, with five of their top ten priorities relating specifically to the social impact of the pandemic, such as the combined impact of the pandemic and the cost of living crisis (see Supplementary Table 1 for all priorities proposed).

Public priorities included in the WCEC work programme covered seven of the 11 themes proposed by the public across a total of 16 projects (see Table 2); 14 evidence synthesis and two primary research. The programme reflected the top public research priorities proposed, including at least one question from each of the top five priority areas (see Table 2). Other themes covered included education and infection prevention and control. Eight of the priorities proposed by the public and accepted on our work programme were also proposed by other stakeholders (e.g. policy, health and social care leads), and were merged (see Table 2 for details).

<<INSERT TABLE 2 HERE>>

Table 2. COVID-19 research priorities proposed by the public and included in the WCEC work programme (n=13)

Meeting UK Standards for Public Involvement

Our public involvement strategy and associated activities (see Table 1) fulfilled all UK Standards for Public Involvement.² A summary of how we addressed each standard is included in Table 3. A key area for improvement is the way in which we captured the impact of public involvement standard.

<<INSERT TABLE 3 HERE>>

Table 3: WCEC public involvement activities mapped to the UK Standards for Public Involvement²

Lessons learnt from involving the public in rapid evidence processes

We collated feedback via quarterly PPG meetings, ad hoc feedback (email and during study meetings), minutes of weekly management meetings, feedback from Collaborating Partners, and learning included in our regular funder progress reports to identify what worked well, key challenges, and possible solutions (see Figure 2 for a summary).

What worked well?

Involvement in all activities across the research-knowledge mobilisation process as a member of the core research team

The type and volume of work and outputs of the PPG (Table 1) showed that it is feasible and advisable to involve the public in all activities across the research-knowledge mobilisation process (see Figure 1 and Table 1), as key members of the research team. Feedback shows that PPG members felt embedded as a core member of the research team, and that their views were considered across all our processes.

“The team are very good open to the exchange of feedback on both sides” [PPG Member]

“I have very much enjoyed working with the WCEC. The highlight was….helping colleagues run a stall at a public venue. There was good co-production with the input into the stall and I was treated as an equal member of the team” [ PPG Member]

Involvement in rapid evidence processes to inform pandemic decision making

Our work shows it is feasible and advisable to involve the public throughout the research-knowledge mobilisation process in a rapid evidence environment. Most of our evidence synthesis projects took two-three months. During this time, the allocated PPG member would be included in 3-4 project meetings with a broad range of stakeholders, contributed to the protocol and the final report, and infographics, and prepared a lay summary. For some projects, members were also involved in disseminating this knowledge via Welsh Government Evidence Briefings and other engagement activities.

We successfully conducted rapid research prioritisation exercises with the wider public, including underserved groups. These consultations took between one- four months from identifying the group, consulting on priorities, collating priorities, assessing priorities for inclusion, allocating included priorities to our work programme, and commencing the work. This ensured the priorities proposed and considered were relevant to the rapidly evolving pandemic context. Twenty-two of the WCEC reports have been cited in Welsh Government publications, thus used to guide policy, advice and / or recommendations.

The important and complementary input of the public in identifying the most important COVID-19 health and care research priorities

Eight of the thirteen priorities proposed by the public and accepted on our work programme aligned closely with what policy makers and practice leads identified as important evidence priorities, demonstrating synergy between what matters to the public and the people making decisions about policy and practice (see ‘merged’ column in Table 2). However, five public pandemic-related research priorities were included in our work programme that were not identified by other stakeholders (e.g. domestic violence impact on younger children, access to care for disabled people, and the long term impact of Long-COVID), highlighting their important and complementary input.

“...I have enjoyed the experience and hope my contribution on priorities has been worthwhile and beneficial for the public in the end” [Workshop participant]

Inclusive and accessible opportunities for public involvement

The ‘Working Together’ needs assessments with PPG members allowed us to tailor involvement opportunities. We were able consider PPG members’ other commitments, personal / health circumstances, and support needs and enable them to participate in ways that worked for them (e.g. holding meetings outside traditional office working hours). Further, our workshops with underserved groups offered flexible, inclusive, and accessible ways to engage participants. By purposively reaching out to those most impacted by the pandemic and offering a choice of virtual versus face-to-face workshops and providing accessibility services, such as sign-language support and speech-to-text services, we were able to recruit a diverse and more representative group of participants.

Virtual involvement

The pandemic restricted typical face-to-face public involvement opportunities. However, we successfully transitioned to virtual platforms and engaged 81 members of the public in a wide range of virtual involvement activities alongside researchers, policy makers, and health and social care leads. Virtual activities ranged from research prioritisation exercises, WCEC governance, research meetings, reviewing documents, contributing to study write-up (papers, reports, and lay summaries), to dissemination of study findings.

Supporting our public partners to develop new skills

Our PPG members reported that the WCEC provided novel ways of working and new opportunities. With guidance from the WCEC team, our public partners learned how to distil the results into digestible and accessible, usually single page lay summaries. As their experience increased, members then felt confident to draft these summaries independently, and then provide mentorship to other members. The administrative team also supported PPG members to navigate the virtual systems, helping to trouble shoot when required. Further, PPG members became involved in tasks that they had not typically been invited to contribute to previously, including defining the target populations and ranking outcomes to be included in the rapid review protocols.

“I have been involved in rapid evidence reviews and that work is different to other work I have done in the past, so it took a while for me to adjust to it….I have written three Lay Summaries. Again this is new….but again I have been developing new skills and experience.” [PPG Member]

“I have learnt much and always felt that my voice was heard…I will always be proud of our work” [PPG Member]

Key challenges and gaps in our work

Accommodating urgent research priorities

The WCEC remit was to rapidly respond to important evidence gaps to inform pandemic decision making in Wales. Frequently the work programme was responsive rather than planned, and the overriding priority was timeliness of evidence for the decision makers. This process typically took 2-3 months, but occasionally we needed to provide evidence within 1-3 weeks, such as the questions around use of face masks and vaccination in pregnant or post-partum women. Meetings needed to be set-up urgently, and the rapidly evolving nature of the pandemic meant information was frequently shared by other involved stakeholders at the last minute. However, this conflicts with best practice guidance for public involvement, which notes public contributors should receive timely updates allowing them to digest information and provide considered feedback. Further, it challenged our ability to fully meet the ‘Inclusive Opportunities’ UK Public Involvement Standard,² by limiting flexibility of when and how the public partners could contribute to these urgent evidence needs.

Balancing the needs of policy, practice, and public stakeholders

Occasionally, it was challenging to balance the needs of all stakeholders involved to agree the specific research question that WCEC could include in its work programme, after the broad evidence need was identified. Some evidence needs, such as long covid, were broad and complex topics that necessitated the involvement of many professional stakeholders with competing needs, not only across health and social services, but also across employment, education, economy, and social justice. Further, some evidence gaps came from under represented and niche areas with limited existing evidence, compared to areas where a robust evidence base already existed. Some PPG members reported in our ‘catch-up’ meeting that they felt like a ‘lone voice’ in complex topics, where each stakeholder had slightly different evidence needs, and when decisions were being made to focus on a more niche and under represented area over a broadly relevant area with an established evidence base. Following this feedback, we reassured our PPG that everyone can contribute to these complex topics and that niche topics are as important as broadly relevant questions and sought to include two PPG members going forward.

Identifying specific research questions from the public with a clear pathway to impact

We successfully involved a diverse group of 99 public members in identifying evidence needs that were subsequently included in our work programme. These represented important evidence gaps. However, it was challenging to identify the specific and answerable research questions that are essential for evidence synthesis, and meet the requirements of ‘good research questions’.¹³ Ideally, research questions should include the essential components outlined in the PICO (population of interest, interventions, control / comparison, and outcomes of interest) or similar framework (depending on the research question type), and also consider the FINER criteria: feasibility, interesting question, novel, ethical and relevant.¹³ Further, evidence needs identified by the public lacked a clear pathway to impact, or in other words, a clear indication of how the results of the work would be translated to make a change in policy or practice, and limited details on the expected impact of this change.

Availability of collaborative and secure platforms for all stakeholders

A consistent request from PPG members was to identify a collaborative online space, where documents could be shared for viewing and editing. Whilst many collaborative platforms exist (e.g. Microsoft Teams, Google Drive, Trello) we were not successful in identifying one platform that could be used by multi University-based research teams, the PPG members and the other stakeholders involved in the research (e.g. NHS or policy representatives). Platforms used by organisations, such as Microsoft Teams, typically require an institutional email address. We also experienced difficulties inter-University and between organisations with this platform. More publicly available platforms, such as Google Drive, were typically blocked by NHS and Government browsers, and did not meet the security standards for sharing sensitive policy-related documents or confidential data from our primary research.

Capturing the impact of public involvement in rapid research

Our data show the impact the public had on determining the questions that were included in the WCEC work programme (13 of the 55 questions proposed were included), including the number of questions that were not proposed by other stakeholders and would have subsequently been missed had we not involved the public (n=5). At the quarterly PPG meetings we encouraged members to share the impact they felt that they had on the projects. However, we did not use a formal mechanism to capture and record this, we did not specifically report public involvement impact across all activities against the separate UK Standards for Public Involvement,² nor did we explore the impact of involvement on the person e.g. improved confidence / understanding of the research cycle – activities we have incorporated into our processes going forward.

<<INSERT FIGURE 3>>

Figure 3: Public Involvement in rapid evidence processes: what worked well, key challenges, and possible solutions

We successfully involved 99 members of the public in identifying COVID-19 research priorities for the WCEC and eight members of the public as research partners in our rapid evidence activities, across the research-knowledge mobilisation process. In a pandemic context, we implemented involvement processes that were inclusive and accessible, utilised virtual platforms and methods, supported the development of new skills for our public partners, and fulfilled the UK Standards for Public Involvement.²Importantly, our findings highlight the essential and complementary input of the public in identifying research priorities, with five questions accepted onto our work programme being distinct to priorities identified by other stakeholder groups. Four of these were proposed by underserved groups hardest hit by the pandemic and included questions about the impact of the pandemic on domestic violence and on existing inequalities experienced by disabled people, further highlighting the importance of including these groups in topic identification. We have learnt much about what worked well, which provides a blueprint for involving members of the public in rapid evidence environments at every stage of the research knowledge-translation process. However, our findings also highlight some important areas for discussion and improvement, including how we capture the impact of public involvement, which we present below.

Balancing robust topic identification processes with pragmatic processes that support rapid evidence turnaround

There are several well-established and robust research prioritisation processes, including James Lind Alliance (JLA) Priority Setting Partnerships (PSPs).⁹ The JLA PSPs provide an excellent method for bringing patients, carers and clinicians together to identify evidence uncertainties and agree which of these matter most and deserve priority attention. The approach also helps to ensure researchers answer questions that can make a difference in routine clinical practice to improve patient care. However, the JLA PSP approach typically takes 12-18 months, and did not align with the rapid evidence requirements we experienced during the COVID-19 pandemic. If we had taken 12-18 months to identify and agree the most important priorities between the public and other stakeholders, the evidence priorities would have been either irrelevant or no longer of use to inform time critical pandemic decision making. We believe our rapid and largely virtual process offers a pragmatic alternative to established long-term prioritisation processes, which also supports the inclusion of policy and practice decision makers in the process and could be particularly useful for rapidly evolving health and care situations.

However, it is important to recognise potential downsides of the rapid approach. It is a more time sensitive exercise, conducted quickly, typically virtually, and less flexibly in terms of timing. This may be particularly problematic from an inclusion point of view. Further, the JLA process offers the added benefit of having all stakeholders involved in one final priority setting stage to agree the priorities.¹⁴ For convenience and inclusion reasons, we conducted the final priority setting stage with the individual public groups and the different stakeholder groups separately, so each had their own list of top ten priorities that were put forward and considered against WCEC criteria for inclusion on our work programme. We found that many of the evidence gaps proposed by the public, and the final priorities each group agreed on, lacked the specificity required and did not tend to consider issues of feasibility. Further, these evidence needs were limited by the lack of information regarding how the evidence might be used to make a change and what the expected impact would be. The JLA approach of involving all relevant stakeholders at each stage may overcome the issues we encountered regarding specificity of the research priorities proposed by the public, and unclear pathways to impact.

Based on our learning, we are not advocating a move to rapid methods over these longer-term processes. Instead, we recommend that a consideration of ‘urgency’ of the priorities could helpfully inform whether to use more pragmatic rapid processes: “if we wait too long to know what the evidence gaps are and to agree on which are most important, will the work still be of use?”.

Do current public involvement best practices align with rapid evidence environments?

Throughout our work, we were aware that the needs of the rapid evidence environment did not always align with public involvement guidelines and best practices. ^E.g. ^2,5,15. To provide an inclusive, supportive, and collaborative public involvement opportunity, we recognise the importance of providing public partners with time to consider their availability for meetings, selecting days and times that take the public partner’s working and social commitments into consideration, offering the opportunity to be sent printed versions of online documents ahead of meetings, balancing in-person and online interactions, and allowing partners the time required to provide meaningful feedback on documents. Even though we were able to involve our public partners in our reviews, it was not always possible to ensure we fully met these guidelines, especially in situations where we had been given a two-week deadline to conduct the research and provide evidence to the policy teams. On reflection, the guidelines do not seem to provide advice specific to rapid research environments, and tend to focus more on longer-term, stable research projects.

In terms of solution, we feel that there should be improved transparency and explanation for these types of opportunities. For example, researchers in rapid environments should be transparent when recruiting to their involvement opportunity about the rapid timeframes, the online nature of collaborating, and identify up-front whether the opportunity will fit with other commitments so that the partner can make an informed decision about participation. They should also provide a clear rationale for the rapid nature and adjusted ways of working, explaining why the opportunity has these requirements and why the rapid nature is important. Further, specific rapid evidence toolkits and training for public members would help to improve public members’ knowledge and skills in this type of research. Of course, when making these changes, we must be mindful of the potential impact on the diversity of public partners we can recruit, and whether we are offering inclusive opportunities. More work is needed to assess the impact of rapid evidence environments on involvement opportunities.

Another area of improvement we identified was formally capturing the impact of public involvement. Since the work was conducted, the Public Involvement in Research Impact (PIRIT) Toolkit¹⁶ has been published and provides a useful framework for researchers to capture and track the difference public involvement makes, allowing researchers to map this directly to the UK Standards for Public Involvement. Further work is needed to understand the feasibility of applying this tool in fast changing rapid evidence environments where the priority is rapid evidence mobilisation, in both evidence synthesis and primary research settings, to explore how we ensure timely completion of the additional administrative tasks required, and how public members themselves could have a role to play in completing part of the tool, whilst not overburdening them.

Since March 2023, we have transitioned from the WCEC to the Health and Care Research Wales Evidence Centre.¹⁷ Our underlying processes and goal of providing timely evidence to Welsh policy and practice leads remain the same, but our topics have evolved from COVID-19 to reach across all health and care. We have taken forward what we know works well in public involvement to the new Evidence Centre, but we have also reflected on the challenges, or what didn’t work so well, and made changes to address these. This includes the introduction of PPG impact logs to capture the impact of public involvement across our processes, enhanced training for PPG members in evidence synthesis (including how we identify answerable research questions), and the opportunity for two PPG members to support each other on broader, longer, and more complex projects, where there are a larger group of stakeholders.

Strengths and weaknesses

We involved a wide range of public members in a variety of ways throughout the research-knowledge translation process, whilst meeting all six UK Standards for Public Involvement,² and public members identified research priorities that were not proposed by other stakeholder groups. Whilst we did take steps in our recruitment and data collection approaches to ensure inclusivity and diversity, there are still some key underrepresented groups that have been excluded, not least because of the predominantly virtual methods required for rapid collaboration. We also recognise that the COVID topic might have provided unique opportunities for involvement, that might not be transferable to other areas of health and care. Unlike other health and social care topics, where some have lived experience of an issue, everyone had lived experience of COVID-19 and was affected in some way, whether it be health, financial, social, wellbeing, education, or employment. This meant that there were many motivated individuals who were eligible for the involvement opportunities, and so our success in recruitment, particularly among underserved communities, occurred in that context.

Our experiences provide a blueprint for involving the public across the research-knowledge mobilisation process in rapid research prioritisation and evidence environments, whilst also highlighting the important and complementary role the public plays in identifying research priorities. Key challenges that require further action include tailored guidance and training for involving members of the public in rapid evidence environments, considering the specific impact of rapid involvement opportunities on research inclusivity and diversity of the participants, and how we can embed processes to capture public involvement impact in contexts where rapid knowledge mobilisation is paramount in a way that does not overburden public partners.

ALPHA: Advice Leading to Public Health Advancement young people advisory group

COVID-19: SARS-CoV-2

EYST: Ethnic Minorities and Youth Support Team Wales

JLA: James Lind Alliance

PIRIT: Public Involvement in Research Impact Toolkit

PPS: Priority Setting Partnerships

PPG: Public Partnership Group

SUPER Group: Service Users for Primary and Emergency Research Group

WCEC: Wales COVID-19 Evidence Centre

Ethics approval and consent to participate

Ethical approval was not required for this work.

Consent for publication

No personal information is presented, and thus consent for publication was not required.

Availability of data and materials

All data relevant to this study are presented in the text or in the supplementary files.

Competing interests

The authors have no competing interests to declare.

Funding

The Wales COVID-19 Evidence Centre was funded by Welsh Government, via Health and Care Research Wales, between March 2021 and March 2023.

Authors' contributions

All authors have contributed substantially to the conception and / or delivery of the public involvement activities outlined in this paper. BH, AS, RK and ND were members of the WCEC Public Partnership Group (PPG). NJW was Public Involvement and Engagement Lead for the WCEC. PG and NJW designed and coordinated the public workshops.

Natalie Joseph-Williams: Conceptualisation, data curation, formal analysis, investigation, methodology, project administration, writing – original draft, writing – review and editing, visualisation, validation

Peter Gee: Conceptualisation, data curation, formal analysis, investigation, methodology, project administration, writing – review and editing

Denitza Williams: project administration, writing – review and editing, validation

Bob Hall: data curation, writing – review and editing, validation

Alexandra Strong: data curation, writing – review and editing, validation

Rashmi Kumar: data curation, writing – review and editing, validation

Nathan Davies: data curation, writing – review and editing, validation

Alison Cooper: writing – review and editing, validation

Ruth Lewis: lead for Evidence Synthesis work programme, overseeing involvement of PPG, writing – review and editing, validation

Micaela Gal: writing – review and editing, validation

Adrian Edwards: project administration, writing – review and editing, validation

Acknowledgements

We are grateful to all members of the public who contributed to the research prioritisation process, including attendees of the four workshops groups who shared their lived experiences of COVID-19 with us. We are grateful to the following organisations for helping us to recruit participants for the workshop: Children’s Commissioner for Wales; the Wolfson Centre for Young People’s Mental Health; Advice Leading to Public Health Advancement (ALPHA) young people advisory group; Taff Housing; Ethnic Minorities and Youth Support Team (EYST) Wales; and Disability Wales.

Health Research Authority and INVOLVE. 2016. Impact of public involvement on the ethical aspects of research: A joint briefing from the Health Research Authority and INVOLVE on how public involvement can help in the ethical design and conduct of research. Available from: https://s3.eu-west-2.amazonaws.com/www.hra.nhs.uk/media/documents/impact-public-involvement-ethical-aspects-research-updated-2016.pdf [accessed 19.04.23]
UK Standards for Public Involvement. 2019. Available from: https://sites.google.com/nihr.ac.uk/pi-standards/home?pli=1 [accessed 19.04.23]
Wales COVID-19 Evidence Centre. Available from: https://healthandcareresearchwales.org/about-research-community/wales-covid-19-evidence-centre [accessed 19.04.23]
Health Research Authority. What is public involvement in research? Available from https://www.hra.nhs.uk/planning-and-improving-research/best-practice/public-involvement/ [accessed 24.08.23]
Health and Care Research Wales. Involving the Public in your Research. Available from: https://healthandcareresearchwales.org/researchers-support-and-guidance-researchers-develop-research-idea/involving-public-your-research [accessed 19.06.23]
Health and Care Research Wales. 2022. Three categories of public involvement. Available from: https://healthandcareresearchwales.org/sites/default/files/2022-10/Three_categories_of_involvement_A4_eng.pdf [accessed 19.04.23]
Erbel M, Joseph-Williams N, Nollet C, Fitzgibbon J, Hatch S. Overcoming the disconnect between scientific research and the public. Immunology and Cell Biology. 2023:101(7);590-597
Evans B, Gallanders J, Griffiths L, Harris-Mayes R, James M, Jones S, Joseph-Williams N, Nettle M, Rolph M, Snooks H, Wallace C, Edwards A. Public Involvement and engagement in primary and emergency care research: the story from PRIME Centre Wales. International Journal of Population and Data Science. 2020;5(3):3.
James Lind Alliance. Reflections on Rapid Priority Setting and the James Lind Alliance Method. 01 June 2023. Available from: https://www.jla.nihr.ac.uk/news/reflections-on-rapid-priority-setting-and-the-james-lind-alliance-method/33585 [accessed 19.10.23]
British Medical Association. The Impact of the Pandemic on Population Health and Health Inequalities. 2022. Available from: https://www.bma.org.uk/advice-and-support/covid-19/what-the-bma-is-doing/the-impact-of-the-pandemic-on-population-health-and-health-inequalities#:~:text=Infection%20and%20mortality%20rates%20have,people%20and%20White%20men%2C%20respectively [accessed 19.10.23]
The Health Foundation. Unequal Impact, Fairer Recovery: The COVID-19 impact inquiry report. 2021. Available from: https://www.health.org.uk/publications/reports/unequal-pandemic-fairer-recovery [accessed 19.10.23]
Wales COVID-19 Evidence Centre. What is the evidence of direct harm from COVID-19 infection and COVID-19 vaccine in pregnant / post-partum women and the unborn child? Available from: https://healthandcareresearchwales.org/what-evidence-direct-harm-covid-19-infection-and-covid-19-vaccine-pregnantpost-partum-women-and [accessed 19.10.23]
Aslam S and Emmanuel P. Formulating a researchable question: a critical step for facilitating good clinical research. Indian J Sex Transm Dis AIDS. 2010;31(1):47-50.
James Lind Alliance. Reflections on Rapid Priority Setting and the James Lind Alliance method. Available from: https://www.jla.nihr.ac.uk/news/reflections-on-rapid-priority-setting-and-the-james-lind-alliance-method/33585 [accessed 19.10.23]
NIHR. Briefing notes for researchers: public involvement in NHS, health, and social care research. 2021. Available from: https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371 [accessed 19.10.23]
Public Involvement in Research Impact Toolkit (PIRIT). 2023. Available from: https://www.cardiff.ac.uk/marie-curie-research-centre/patient-and-public-involvement/public-involvement-in-research-impact-toolkit-pirit [accessed 19.10.23]
Health and Care Research Wales Evidence Centre. Available from: https://researchwalesevidencecentre.co.uk/ [accessed 19.10.23]

Tables 1 to 3 are available in the Supplementary Files section

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Involving the public in research prioritisation and as research partners in the Wales COVID-19 Evidence Centre (2021-23): processes, research priorities, outputs, and lessons

Status:

Version 1

Abstract

Figures

Plain English Summary

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1