Engaging citizens in the co-production of a health system performance assessment framework: a case study in Ireland

doi:10.21203/rs.3.rs-605225/v1

Download PDF

Research

Engaging citizens in the co-production of a health system performance assessment framework: a case study in Ireland

https://doi.org/10.21203/rs.3.rs-605225/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 19 Dec, 2021

Read the published version in Health Research Policy and Systems →

You are reading this latest preprint version

Background

The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework, which should reflect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reflects on the added value of using a citizen panel in the co-production of an HSPA framework.

Methods

A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panelists received lay-language preparatory materials prior to the meeting. Panelists used a three-color scheme to signal the inclusion and importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups.

Results

The citizen panel generated 249 health performance measures ranging across 13 domains. Domains assessed as the most important included people-centeredness, coordination of care, and coverage. Prioritization of domains differed between panels. Citizen panelists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative.

Conclusions

The engagement of citizens early on in the co-production process of the HSPA framework shaped the processes that followed, with the restating of priorities of the citizen panel informing decision-making throughout. Citizen engagement in HSPA development is essential for realizing value-based people-centered health systems and assuring an inclusive process that helps to generate trust and ownership of performance intelligence. Future research could expand on how citizen panels could be further engaged in co-creating mechanisms to assess, monitor, and report on the performance of health care systems.

Health Economics & Outcomes Research

Health Policy

Public deliberation

Priority setting

Focus group

Stakeholders panel

Health care system performance

Performance intelligence

Ireland

Sláintecare

Health system performance measurement, and its use as performance intelligence, plays a central role in guiding the decisions of health system actors towards improved outcomes [1] and care experiences [2]. Publicly reporting performance measurement can also contribute to encouraging improvements [3], as well as system transparency and accountability [1, 4]; the latter contributing to public trust and a sense of ownership of the health care system [5]. Of the varied approaches to measure performance [1], the use of Health System Performance Assessment (HSPA) is a well-established and widely-supported [6, 7] means to evaluate the health system as a whole. HSPA is a country-specific participatory process, relying on a limited number of indicators to link outcomes with system functions or strategies [8]. In the European context, the development of tools and methodologies to support performance measurement [9–11] and a range of country studies on the use of HSPA [12–15] have served to shift discussions from why to use an HSPA approach to focus on the what and how of the process.

Citizen engagement processes can take on varied forms: citizen panels, focus groups, citizen juries, consensus conferences, or citizen assemblies [16, 17]. Importantly, these methods are distinct from more passive approaches focused on preference elicitation, such as polling and surveying. In contrast, citizen panels put emphasis on empowering citizens to actively deliberate in processes that precede policy-making [18]. This contribution is presumed to have important advantages to supporting ownership, legitimacy, and fairness in priority-setting processes, as well as building public trust and greater acceptance in decisions [5, 19].

In Ireland, improving the governance, accountability, and performance of the health system was set out as a key priority in the 10-year reform programme Sláintecare 2019–2028 [20]. At the outset of the programme, key stakeholders recognized that a comprehensive performance measurement framework and management system was needed to foster accountability and capture achievements against the objectives of Sláintecare [21]. To this end, the Department of Health of Ireland (DoH) requested the support of the European Commission’s Structural Reform Support Service and, in mid-2019, the project “Performance Accountability for the Irish Health System” was launched. A team of Health Care Performance Intelligence Professionals from Amsterdam University Medical Centers was selected to support this project. Early in the project, priority was given to selecting a set of domains and performance measures that reflect shared priorities of multiple stakeholders, including citizens.

While methods for engaging stakeholders are well-established in their own right [22, 23], citizen participation applied to HSPA processes has yet to be embedded into development cycles [15, 24]. The engagement of citizens in participatory policy processes such as the development of an HSPA framework links to the broader discussion on value-creating learning health systems [25]. Being considerate of the central role of citizens’ care experiences and perceptions about the health care system can strengthen value creation and nurture public trust and a sense of ownership [5]; hence, contributing to the health care system becoming more people-centered and prioritizing care and outcomes that matter most to patients [26]. For example, the International Consortium for Health Outcomes Measurement has recognized the importance of participatory processes by engaging patients and health care professionals in developing patient-reported outcomes measures (e.g., [27]). Alike to these participatory processes, with this study we set out to apply methods for the engagement of both citizens and other stakeholders towards the development of Ireland’s first HSPA framework. This study aims to describe the method applied in the Irish context and to reflect on the added value of using a citizen panel in the co-production of an HSPA framework.

Design

This study focuses on the use of a citizen panel conducted in December 2019, acting as representation of the Irish public in the scope of developing an HSPA framework for Ireland. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including: 1) representatives of the DoH and Health Service Executive (HSE) – internal stakeholder panel; and 2) representatives from key professional associations and special interest groups such as the academia and patient organizations – external stakeholder panel (Fig. 1). Approximately, 30 people identified by the DoH and HSE participated in each of those panels. The citizen panel included 15 people of varying walks of life. Participants in the citizen panel were compensated for their travel and received a stipend for their time; they also signed an informed consent prior to engaging in the meeting and were aware that they could withdraw from participating in the session at any time.

Literature review

At the outset of the study, a targeted review of the scientific and grey literature on HSPA was conducted to consolidate a list of potential themes (domains) for the framework being developed in Ireland. We took as a basis an international comparative study of HSPA domains applied across the World Health Organization European Region’s 53 member states [12], as well as an internal review by the DoH. The list of domains (health and well-being, equity, accessibility, quality of care, social and financial risk protection, coverage, safety, responsiveness, efficiency, effectiveness, people-centered, continuity of care, and coordination of care) and their definitions were consolidated for use in the panels that followed.

Citizen panel

Sample and recruitment

A target of 15 panelists were pursued based on prior experiences with the use of citizen panels in the health sphere [28]. Citizens were recruited by a Dublin-based company providing recruitment services for qualitative research. The definition of a citizen eligible to participate in the panel was that of a lay person, resident of Ireland, who was not professionally involved with the health care system, nor a public official. The sampling strategy aimed to achieve diversity among participants, taking into consideration eight factors: representation of different counties; sex; age; highest level of education attained; nationality; ethnicity; health status; and religious beliefs. A recruitment scheme was developed based on the key characteristics of the population according to 2016 census data [29] (Additional file 1). The recruitment agency reached prospective panelists by phone through an established registry. A screening questionnaire was applied which informed citizens on the context of this work and assessed the diversity factors to ensure the targeted sample was achieved (Additional file 2).

Workshop

A one-day citizen panel workshop took place in December 2019. A lay language citizen brief and workshop programme – both revised by the DoH for clarity of language – were prepared and shared with panelists via email one week before the workshop (Additional file 3). Two moderators (EB and OBF) led the panel’s work and one notetaker (DK) took detailed notes throughout the day. A letter of consent was completed at the outset of the workshop. The day’s activities were organized in three blocks. First, the basis for convening the citizen panel, as detailed in the citizen brief, was restated to ensure this context was understood together with the aims of the panel. Second, the preferences of citizens on what information should be measured and reported on Ireland’s health care system was collected through a prioritization exercise. Third, citizens were tasked with identifying priority approaches and channels whereby information on the health care system could be reported back to the public.

In the scope of the second block aforementioned, we proceeded as follows: first, the panel moderators summarized the most commonly used domains in other countries to report on the performance of their health care systems. After the presentation of a domain, citizens were asked to write down on sticky notes what measures/topics were of concern to them. The color scheme sought to capture the degree of importance assigned to a measure/topic by a panel participant (red: extremely important; yellow: important; and green: somewhat important). When all domains were discussed, the workshop moderators grouped the sticky notes on a whiteboard, producing a heat map used to clarify priorities. At the closing of the workshop, participants were asked to complete an anonymous questionnaire that asked about: 1) the panelist’s experience in taking part in the workshop; 2) how well prepared the panelist felt to take part in the panel; 3) the completion of a projective-inspired task [30] by completing the sentence “Citizen panels are…”; and 4) aspects related to the panelist’s experience or comments regarding the organization of the event (Additional file 4).

Data analysis

The research team assigned the colored sticky notes produced during the workshop to the HSPA domains. Data from the prioritization exercise with the citizen panel were consolidated in a ranking chart. Thereafter, we compared the ranking of domains of the citizen panel relative to that consolidated during the internal and external stakeholder panels. These panels followed a similar structure to that of the citizen panel. The four-question exit questionnaire applied to the citizen panel participants was analyzed in full. To improve the validity of the results, we triangulated data from multiple sources [31], including notes from the panel sessions, feedback from the moderators, experience sharing and discussions among the research team, and questionnaire data. The latter was particularly valuable to synthesize the experience of citizens during the panel, using their own words.

Characteristics of the participants

A total of 15 participants (7 men and 8 women) participated in the citizen panel (Table 1). The age group distribution was as follows: 18–24 (n = 2); 25–34 (n = 4); 35–44 (n = 1); 45–54 (n = 4); 55–64 (n = 1); and 65 and over years (n = 3). The highest educational level for one participant was Junior Certificate; four participants attained a Leaving Certificate, three a Certificate/Diploma, six a Bachelor diploma, and one participant had a Graduate/Master’s certification. Most participants had a paid job (n = 7), reported being in good health (n = 11) and lived in urban areas (n = 14). The nationality of most respondents was Irish (n = 13), including one of Asian descent, and two had nationalities of other European countries (Croatia and Italy) but resided in Ireland.

Table 1

Citizen panel participants’ characteristics
	n	%
Citizen panel	15	100
Sex
Female	8	53
Male	7	47
Age (years)
18–24	2	13
25–34	4	27
35–44	1	7
45–54	4	27
55–64	1	7
65+	3	20
Education
Primary	4	27
Secondary	4	27
Tertiary	7	47
Employment status
With a paid job	7	47
Without a paid job	3	20
Student	2	13
Retired	3	20
Health status (Good health)	11	73
Location (Urban)	14	93
Nationality
Irish	13	87
Italian	1	7
Croatian	1	7

Citizen panel prioritization of HSPA domains

During the panel session, 249 colored sticky notes were used by participants with suggestions of health system performance measures: 153 were marked as extremely important, 79 as important, and 17 as somewhat important (Fig. 2). The domains with most measures proposed were related to people-centeredness (n = 47), coordination of care (n = 31), coverage (n = 29), health and well-being (n = 27), and equity (n = 24); within these, most of the measures were considered ‘extremely important’.

We observed great variation in the priority assigned to each domain by the citizen panel compared to the prioritizations assigned by the internal and external stakeholder panels (Fig. 3). Specifically, the top-ranking domains to the citizen panel (people-centeredness, coordination of care, and coverage) were less prioritized by the internal and external stakeholder panels; domains that were less prioritized by the citizen panel, such as accessibility, responsiveness, efficiency, and effectiveness, were of higher priority in the other panels. ‘Health workforce’ was suggested as a standalone domain by both the internal and external stakeholder panels, whereas in the citizen panel the discussion was brief and centered on the inefficiencies of the workforce working environment (e.g., paper workload).

Citizen panel participants’ experiences

Participants described their experience during the citizen panel as enjoyable, interesting, and informative. The citizen panel was depicted as an opportunity to discuss and develop a broader understanding of the functioning of Ireland's health care system by the experiences and viewpoints of other participants, as highlighted by two of the panelists:

“I am more aware now on how the health system works through others.” (Panelist #15)

“[Participating in the citizen panel was] Very interesting and informative. Did not realise how difficult it was for people going through the public system. Hardship on public system seems much bigger that shared across social media.” (Panelist #3)

All participants appeared well prepared for the panel work; this was highlighted by the participants themselves, who mentioned that the citizen brief was sufficient for their preparation, as suggested by one participant: “Not an issue with pre-reading materials and guidance of facilitators.” (Panelist #11) Most of the participants described that they relied on their experiences with the health care system to engage in the panel's discussions; hence, feeling confident on their preparation for the panel work, as highlighted by a panelist: “[I felt] Very well prepared as the only thing requested was our own experience.” (Panelist #9). Other participants prepared themselves differently atop of the citizen brief: “Done quite a lot preparation in the past week, including to ask others on their views of the health service and what they thought to be important to improve it.” (Panelist #12) It was also noted that the use of lay language at all times during the panel session, including any written materials (e.g., citizen brief), was appreciated: “(…) [panel moderators] were very easy to follow and listen to” (Panelist #8), and thus, affected positively the participants’ experiences.

In general, participants showed similar social representations of what a citizen panel entails, describing it as “[a method of] eliciting info from different ideas/experiences from a diverse group,” (Panelist #11) and “(…) allowing debate to summarise and retain the opinions and ideas of the citizens into more actionable and basic concepts and thoughts,” (Panelist #5), which could be “(…) very important for people to be able anonimously [sic] express their beliefs and talk about their experience in hope of a change for better.” (Panelist #9)

We described the methods of using a citizen panel in the co-production of an outcome-oriented HSPA framework for Ireland. The citizen panel prioritized more people-centeredness aspects of performance domains compared to the internal and external stakeholder panels. This prioritization by the citizen panel is aligned with an interpretation of the Irish Health Survey 2019 data, which suggested the need for reform of the health system to achieve greater value, performance, and person-centered quality of care [32]. To achieve such goals, it is key to include the citizens’ views to steer the health system to better address the populations needs, expectations, and preferences.

Citizen panels are generally underutilized, despite its growing use in recent years in response to many societal challenges [17]. Factors that may be contributing to this underutilization are the limited empirical evidence concerning the effects of such panels, in addition to their time-consuming, labor-intensive, and costly nature, relative to other traditional approaches to elicit the priorities and preferences of citizens such as surveys. To our knowledge, the use of a citizen panel in the scope of the co-production of an HSPA framework is novel. The engagement of citizens early on in the co-production process shaped the processes that followed, with the restating of priorities of the citizen panel informing decision-making throughout. Following principles of inclusiveness in governance [6, 9, 33] and in creating value in health care [26, 34], by involving citizens we actively contributed to fostering the legitimacy, responsiveness, and accountability of the framework in safeguarding the interests of a broader range of stakeholders. This was discussed during project meetings with the International Advisory Board – a consultive group of HSPA experts from different European countries – which welcomed the inclusion of the citizen panel in the process of developing Ireland’s HSPA framework. We think that including citizens in this process was possible in Ireland given the democratic culture of listening to the citizens’ voice during policy-making cycles. In other countries where citizens experience less empowerment or engagement in policy-making processes, using a similar approach to develop an HSPA framework may prove (more) challenging.

In general, citizen panelists shared a similar understanding of what a citizen panel involves. This could signal an effect of the citizen brief used to communicate with participants prior to the panel, where we sought to address any information asymmetry by clarifying in lay language what a citizen panel was and presenting the key themes that served as a basis for discussions during the citizen panel workshop. We noted that most of the citizens came well prepared and were fully engaged in the discussions. This was contrary to our initial expectations, where we feared that health literacy levels in Ireland [35] and the participants’ ability to think outside their personal experiences could hinder a full participation. Most panelists also considered their participation as part of their civil duty, particularly in terms of representing other fellow citizens. This was aligned with our views on the advantages of using the citizen panel in the co-production of the framework, where we considered citizen panel participants as proxies to other people alike. Hence, our focus was not on a panelist’s personal experiences per se, rather about developing a collective perception of the experiences and preferences of the citizen panel as a whole.

Strengths and limitations

The key strengths of our study are the participatory and inclusive characteristics of the process of co-producing the first HSPA framework for Ireland by including three stakeholder panels (including a citizen panel), with the advantage of comparing prioritizations among panels, and the profile of the teams involved during the stages of developing the framework, bringing technical expertise but also having close collaboration with local experts and decision-makers. We recognize that using only one citizen panel limits any ambition to achieve representativeness of Ireland’s population, and that different insights could have been generated, had we organized multiple citizen panels. However, organizing multiple panels would have not been possible due to time and cost constraints, notwithstanding, we strove to reach sufficient diversity with the citizens recruited to participate in the panel, relative to the general population of Ireland. Another limitation relates to the difficulty of controlling all aspects related to the process of recruiting panelists, either a database is already established from which a sampling strategy could be employed, or else commissioning the recruiting of panelists to a local agency is probably best.

In the context of Ireland’s Sláintecare health reform, strengthening participatory citizenship and end-to-end accountability on the performance of the health care system has become a topic of interest to a broader audience; hence, the intent of strengthening the voice of citizens as part of shared decision-making in the co-production of an HSPA framework. The engagement of citizens early on in the co-production process of the HSPA framework permitted the restating of the panel’s priorities in the processes that followed. Following principles of inclusiveness in creating value in health care, citizens’ engagement in the development of an HSPA framework is a crucial contribution for realizing value-based people-centered health systems. It contributed to public trust and a sense of health system ownership of which the importance has only been enforced over the period of the COVID-19 pandemic. Although the HSPA is a country-owned process, it builds-off of international practices and experiences. Other countries that seek to strengthen value creation in their health systems via the engagement of citizens in HSPA development cycles, could draw from these experiences and learnings in Ireland. Future research could expand on how citizen panels could be further engaged in co-creating mechanisms such as HSPA to assess, monitor, and report on the performance of health care systems.

HSPA: Health System Performance Assessment

DoH: Department of Health

HSE: Health Service Executive

VBHC: Value-based health care

Ethics approval and consent to participate

Ethical approval is not applicable. Before taking part in the citizen panel, participants signed a consent form giving permission to use their participation data for the analysis.

Consent for publication

Not applicable.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Competing interests

The authors declare that they have no competing interests.

Funding

The participation of DI, EB, and OBF occurred within a Marie Skłodowska-Curie Innovative Training Network (HealthPros – Healthcare Performance Intelligence Professionals) that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement Nr. 765141 (https://healthpros-h2020.eu). The participation of DK, NK, and TJ occurred within the European Commission Structural Reform Support Service project SRSS/C2019/046 – “Performance accountability for the Irish health system.” The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Authors' contributions

DK and NK led the study conceptualization and supervision. All authors were involved in data collection and analysis. All authors were involved in writing, revising and editing the manuscript, which final version was read and approved by all authors.

Acknowledgements

We acknowledge the contributions and are grateful for the following: the time and input of citizen and stakeholder panel participants, technical reviews and local coordination by Robert Mooney (Department of Health), and methodological support of Kerry Waddell (McMaster Health Forum).

Smith P, Mossialos E, Papanicolas I: Performance measurement for health system improvement: experiences, challenges and prospects. WHO Regional Office for Europe, 2008.
Kuluski K, Nelson MLA, Tracy CS, Alloway CA, Shorrock C, Shearkhani S, Upshur REG: Experience of Care as a Critical Component of Health System Performance Measurement: Recommendations for Moving Forward. Healthc Pap 2017, 17(2):8-20.
Campanella P, Vukovic V, Parente P, Sulejmani A, Ricciardi W, Specchia ML: The impact of Public Reporting on clinical outcomes: a systematic review and meta-analysis. BMC Health Services Research 2016, 16(1):296.
Papanicolas I, Smith P: Health system performance comparison: An agenda for policy, information and research Maidenhead: McGraw-Hill, 2013.
OECD: Trust and Public Policy: How Better Governance Can Help Rebuild Public Trust. OECD Publishing, 2017.
WHO Regional Office for Europe: The Tallinn Charter: Health Systems for Health and Wealth. WHO Regional Office for Europe, 2008.
Paoli F, Schmidt I, Wigzell O, Ryś A: An EU approach to health system performance assessment: Building trust and learning from each other. Health Policy 2019, 123(4):403-407.
Kickbusch I, Gleicher D: Governance for health in the 21st century In. Copenhagen: WHO Regional Office for Europe, 2012.
WHO Regional Office for Europe: Health systems performance assessment: a tool for health governance in the 21st century. WHO Regional Office for Europe, 2012.
Kelley J, Hurst J: Health care quality indicators project conceptual framework paper. In: OECD Health Working Papers No 23. OECD Publishing, 2006.
Expert Group on Health System Performance Assessment: Tools and methodologies to assess the efficiency of health care services in Europe. European Union, 2019.
Fekri O, Macarayan E, Klazinga N: Health system performance assessment in the WHO European Region: which domains and indicators have been used by Member States for its measurement? In: WHO Health Evidence Network Synthessi Report 55. WHO Regional Office for Europe, 2018.
Braithwaite J, Hibbert P, Blakely B, Plumb J, Hannaford N, Long JC, Marks D: Health system frameworks and performance indicators in eight countries: A comparative international analysis. SAGE Open Medicine 2017, 5:2050312116686516.
WHO Regional Office for Europe: Case studies on health system performance assessment: a long-standing development in Europe. WHO Regional Office for Europe, 2012.
Noto G, Corazza I, Klavina K, Lepiksone J, Nuti S: Health system performance assessment in small countries: The case study of Latvia. Int J Health Plann Manage 2019, 34(4):1408-1422.
Mann C, Voss J-P, Amelung N, Simons A, Runge T, Grabner L: Challenging futures of citizen panels: Critical issues for robust forms of public participation. A report based on an interactive, anticipatory assessment of the dynamics of governance instruments. Technische Universität Berlin, 2014.
OECD: Innovative Citizen Participation and New Democratic Institutions. OECD Publishing, 2020.
What’s the difference between deliberation and participation? [https://www.citizenlab.co/blog/civic-engagement/whats-the-difference-between-deliberative-and-participatory-democracy/]
Bruni RA, Laupacis A, Martin DK: Public engagement in setting priorities in health care. CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne 2008, 179(1):15-18.
Department of Health: Sláintecare Action Plan 2019. Department of Health, 2019.
Burke S, Barry S, Siersbaek R, Johnston B, Ni Fhalluin M, Thomas S: Slaintecare - A ten-year plan to achieve universal healthcare in Ireland. Health Policy 2018, 122(12):1278-1282.
Gauvin F: Citizen Panel program. McMaster Health Forum, 2017.
Holmes L, Cresswell K, Williams S, Parsons S, Keane A, Wilson C, Islam S, Joseph O, Miah J, Robinson E et al: Innovating public engagement and patient involvement through strategic collaboration and practice. Res Involv Engagem 2019, 5:30.
Nuti S, Noto G, Vola F, Vainieri M: Let’s play the patients music: A new generation of performance measurement systems in healthcare. Management Decision 2018, 56(10):2252-2272.
Menear M, Blanchette MA, Demers-Payette O, Roy D: A framework for value-creating learning health systems. Health Res Policy Syst 2019, 17(1):79.
Porter M, Teisberg E: Redefining health care: creating value-based competition on results. Harvard Business School, 2003.
Kim AH, Roberts C, Feagan BG, Banerjee R, Bemelman W, Bodger K, Derieppe M, Dignass A, Driscoll R, Fitzpatrick R et al: Developing a Standard Set of Patient-Centred Outcomes for Inflammatory Bowel Disease-an International, Cross-disciplinary Consensus. J Crohns Colitis 2018, 12(4):408-418.
Mattison C, Wilson M, Gauvin F: Panel summary: Optimizing patient and family transitions from cancer treatment to primary- and community-care supports in Canada. McMaster Health Forum, 2018.
People and Society [https://www.cso.ie/en/statistics/]
Oppenheim A: Projective techniques in attitudes study. In: Questionnaire design, interviewing and attitude measurement. edn. London: Continuum, 1966/2001: 210-228.
Leung L: Validity, reliability, and generalizability in qualitative research. J Family Med Prim Care 2015, 4(3):324-327.
Irish Health Survey 2019 [https://www.socialjustice.ie/content/policy-issues/irish-health-survey-2019]
WHO Regional Office for Europe: Pathways to health system performance assessment: A manual to conducting health system performance assessment at national or sub-national level. WHO Regional Office for Europe, 2012.
Smith P, Sagan A, Siciliani L, Panteli D, McKee M, Soucat A, Figueras J: Building on value-based health care: towards a health system perspective. In. Edited by World Health Organisation, European Observatory on Health Systems and Policies. World Health Organisation, 2020.
Marshall S, Sahm L, McCarthy S: Health literacy in Ireland: reading between the lines. Perspect Public Health 2012, 132(1):31-38.

Additionalfile1.pdf
Additional file 1: Citizen panel recruitment scheme and considerations
Additionalfile2.pdf
Additional file 2: Citizen recruitment screening questionnaire
Additionalfile3.pdf
Additional file 3: Citizen brief and workshop programme
Additionalfile4.pdf
Additional file 4: Citizen panel exit questionnaire

Download PDF

Journal Publication

published 19 Dec, 2021

Read the published version in Health Research Policy and Systems →

Editorial decision: Major revision
14 Oct, 2021
Review #2 received at journal
13 Oct, 2021
Reviewer #2 agreed at journal
05 Oct, 2021
Review #1 received at journal
30 Aug, 2021
Reviews received at journal
15 Jul, 2021
Reviewer #1 agreed at journal
14 Jul, 2021
Reviewers invited by journal
12 Jul, 2021
Editor assigned by journal
05 Jul, 2021
Submission checks completed at journal
05 Jul, 2021
Editor invited by journal
05 Jul, 2021
First submitted to journal
05 Jul, 2021

You are reading this latest preprint version

Engaging citizens in the co-production of a health system performance assessment framework: a case study in Ireland

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Methods

Design

Literature review

Citizen panel

Sample and recruitment

Workshop

Data analysis

Results

Characteristics of the participants

Citizen panel prioritization of HSPA domains

Citizen panel participants’ experiences

Discussion

Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Journal Publication

Version 1