Development of the Patient and Public Involvement Questionnaire (PPIQ) for Canadian Drug Funding Committees

doi:10.21203/rs.3.rs-23414/v1

Download PDF

Research article

Development of the Patient and Public Involvement Questionnaire (PPIQ) for Canadian Drug Funding Committees

https://doi.org/10.21203/rs.3.rs-23414/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background There is limited evidence evaluating the impact of patient and public involvement in setting priorities for healthcare, specifically drug funding recommendation committees. We describe the development phases of the Patient and Public Involvement Questionnaire (PPIQ).

Methods The development of the PPIQ was informed by previous work, which established nine criteria to evaluate patient and public involvement. The PPIQ was developed with a multi-method, multi-phased approach: 1) item generation and refinement (item bank creation, user feedback sessions); 2) sensibility testing (using Feinstein’s criteria and structured interviews); and 3) pilot testing with drug funding committee’s in Canada.

Results In phase one of development, a bank of 846 items were derived from key informant interviews and a literature review. Guided by the nine evaluation criteria, an initial draft of the PPIQ was created using the item bank. Two user feedback sessions (n=7) resulted in further revisions of the PPIQ. In phase two, participants (n=21) completed a sensibility questionnaire with a median score 6 out of 7 on 80% of items. Interview participants (n=14) articulated the PPIQ was clear and appropriate. In phase three, response rate for pilot testing the PPIQ was 25% (n=14) and the average time for participants to complete the PPIQ was 00:19:00 minutes (SD ± 13:46).

Conclusions The methods used for the development of the PPIQ including qualitative interviews, literature review, user feedback, sensibility and pilot testing have resulted in a questionnaire that may be used with committees making drug funding recommendations in Canada.

Health Economics & Outcomes Research

Health Policy

Patient and public involvement

public drug funding

reliability and validity

sensibility

There has been a growing global commitment by policy-makers and providers to improve the quality of health-related decision making by incorporating greater patient and public involvement (1–5). However, despite this increase in stakeholder involvement, there is limited evidence of how this involvement affects health care priority setting (6,7).

Evidence has shown that issues raised by patients and patient advocacy groups are reflected in drug-related health technology assessment recommendations and hospital-based assessments (8,9). This evidence demonstrates that patients and patient advocacy group perspectives were often considered and that they enhanced the content and framing of the recommendations (8,9). However, inclusion of patient insights is only one facet of measuring meaningful participation; it does not address for example, the nature and quality of their participation in the process of committee deliberation (8,9). Evaluating the effect of patient and public involvement poses several challenges. These include determining and evaluating “successful” involvement, accounting for the effect of power differentials between different types of experts, and determining whether resources to support involvement are sufficient (10).

The aim of this paper is to describe the development of the Patient and Public Involvement Questionnaire (PPIQ), a tool to evaluate the degree of involvement of all committee members, including patients and members of the public, in decision making within the specific context of Canadian committees that make formulary listing recommendations. Our phases included the following: 1) generating and reducing PPIQ items (i.e., questions); 2) assessing sensibility, and 2) pilot testing of the PPIQ.

We used both quantitative and qualitative methods to develop, refine, assess, and pilot the PPIQ with stakeholders involved in Canadian formulary recommendation processes (Fig. 1). Consistent with the principles of the process, we included the perspectives of committee members, patient group representatives, public drug plan employees and academic experts (national/international) in patient and public involvement in healthcare.

Conceptual Approach

The development for the PPIQ builds upon previous work by authors ZRY and AMB who developed nine criteria to evaluate patient and the public involvement in health care resource allocation decision making (10). In this work, authors ZRY and AMB conducted a literature review of studies describing the development or evaluation of questionnaires to measure public or patient involvement in public decision making (including non-health related fields). Additionally, the authors conducted key informant interviews with representatives of patient groups, past or present government employees, representatives from Canadian provincial Ministries of Health, advisory committee members and industry personnel. These nine criteria, outlined in Table 1, guided the refinement and organization of PPIQ throughout development.

Table 1

Nine Evaluation Criteria of Public and Patient Involvement (10).
Evaluation Criteria	Description
Clarity Regarding Rationale and Roles of Patient and Public Members	The purposes of patient and public involvement should be clearly defined. This criterion may define whether a member’s role is to represent the broader public view, the patient view, or both. Furthermore, rationale may define whether the objective of involvement is to consult with external patient advocacy groups or to actively partner with them in making decisions.
Adequate Representation of Relevant Views	A full range of perspectives should be represented. This criterion may assess whether patient and public members are sufficiently representative of their constituencies and whether patient advocacy group submissions that are received from several groups adequately represent diversity.
Fair Decision-Making Processes	The processes and procedures used by the committee should ensure fairness in decision making from the perspective of patient and public representatives.
Sufficient Support	Patient and public representatives should have sufficient support to fulfill their responsibilities.
Legitimacy of Committee Processes	Committees’ processes should be accountable and transparent. This criterion may assess whether participants understand how decisions are made and whether stakeholders are able to appeal decisions
Adequate Opportunity for Participation	Committees should have processes that allow public and patient representatives the opportunity to participate fully in discussions and to address power differentials.
Meaningful Degree of Participation	Committee deliberations should include meaningful patient and public participation. This criterion may assess the level of participation of public and patient members in decisions and the number and quality of patient advocacy group submissions.
Noticeable Effect on Decisions	Public and patient involvement should materially influence committees’ recommendations. This criterion may assess how many decisions would have led to different conclusions had patient or public involvement not been present
Considerations of Efficiency	Committees should have sufficient resources to ensure effective participation, which should be used efficiently. This criterion may assess the time and money needed to make recommendations and the value obtained relative to the resources expended.

Phase 1: Item Generation And Reduction

1.1 Item Bank. The literature review and key informant interviews (methods described in detail: Rosenberg-Yunger & Bayoumi, 2017) were used in the current work to inform the first phase of development for the PPIQ. The interviews and literature review provided a dataset that we used to develop items. We coded the dataset in an inductive line-by-line manner. We considered each code individually to develop the preliminary list of items (i.e, the item bank). Two members of the research team (ZRY and LV) reviewed the item bank to reduce the number of redundant items, identify missing items, and ensure items adequately addressed the purposes of evaluating patient and public involvement based on the nine criteria of public and patient involvement.

1.2 User Feedback (Focus Group). We conducted one in-person focus group discussion with Canadian Agency for Drugs and Technologies in Health (CADTH) staff, representatives from patient groups, members of drug advisory committees, government employees, and industry personnel to capture stakeholder feedback on the PPIQ (Version 1). We identified informants from websites listing drug advisory committee members, and suggestions by staff in Ministries of Health. Written informed consent was received from all participants either in person or electronically.

Focus group participants were asked to review the draft PPIQ (Version 1) eliminate redundant items, add missing items, and ensure each of the nine criteria of public and patient involvement was adequately captured in the questionnaire. Additionally, we asked participants to assess ambiguity and to check for double-barreled items. Conflicting views on which items to retain or delete were addressed at this stage by retaining these items. The final decision to retain or delete items was made after the focus group data was analyzed. The focus group discussion was audio-recorded and transcribed. Two team members (ZRY and LV) analyzed the focus group transcript independently using a qualitative thematic approach resulting in suggestions to add, retain or remove items from the PPIQ. Any discrepancies in coding were discussed and resolved through consensus. We revised items based on recommendations from the focus group to result in a refined PPIQ (Version 2).

1.3 User Feedback (Online Survey). A second group of participants, similar to the first group, provided feedback on the PPIQ (Version 2) by answering questions about questionnaire refinement (including item generation and reduction) in an online survey using Opinio (Copyright 1998–2019 ObjectPlanet). Participants were also asked to provide electronic feedback by reviewing items within the PPIQ as a Microsoft Word document, which was emailed to each participant. Participants were asked to make comments and track changes throughout the PPIQ Word document. Two team members (ZRY and LV) coded the open-ended survey responses and discussed and resolved any discrepancies in coding through consensus. The comments and track changes by each participant helped inform further refinement of the PPIQ not captured in the survey responses. We revised the PPIQ based on the online surveys and electronic feedback to create PPIQ (Version 3).

Phase 2: Sensibility Testing

2.1 Sensibility Questionnaire. A total of 115 emails were sent out, of which 99 were delivered to stakeholders (35 committee members, 24 patient groups representatives and 40 academic experts) requesting their participation in the sensibility testing and 21 participated. The average age of participants was 50.9 years (standard deviation ± 12.3), and the majority identified as male (n = 13, 62%). Six of the participants were members of drug reimbursement committees, of whom five were professional members and one was a public member. Eight participants were patient group representatives, and four were academic experts in the field of patient and public engagement. Three participants remained anonymous.

Twenty-one participants reviewed the PPIQ and completed the sensibility questionnaire. The five analyzed categories, including purpose and framework, overt format, face validity, content validity, and ease of use, were reflected by 9 items in the sensibility questionnaire. Table 3 provides a summary of responses by all participants and also by the different categories of participants. The median scores on the sensibility questionnaire were ≥ 5 (out of 7) for all 9 items (100%) reaching our criteria for sensibility. Three participants were anonymous and did not provide their role on committees. Only one public member completed the sensibility testing. Overall, the item with the lowest score (5 out of 7) was the participants’ perception of the time to complete the PPIQ (Table 4).

Table 2

Feinstein’s Sensibility Framework (12)
Criterion	Definition
Purpose and framework	Considers issues pertaining to function as well as applicability of the questionnaire. Examples include: what is the purpose of the questionnaire?
Overt format	Considers issues pertaining to the manner in which the questionnaire is presented.
Face validity	Determines whether items are consistent with the purpose of the questionnaire, assesses if items are clear, directed at the correct person and if overall the questionnaire is reasoned
Content validity	Considers the selection of items including missing item, and inclusion of unsuitable items, as well as, appropriateness of response options.
Ease of use	Considers the ease of administration and use of the questionnaire from the users and researchers’ perspectives.

Table 4

Summary of responses for sensibility questionnaire (n = 21 participants)
Sensibility Questions	Sensibility Framework Criteria	All Participants Median Responses* (n = 21)	Patient Group Members (n = 8)	Professional Committee Member (n = 5)	Academic/Expert (n = 4)
1) Please rate the questionnaire (PPIQ) in terms of clarity and simplicity.	Face validity	6	6	5	5
2) Were the questionnaire (PPIQ) instructions adequate?	Overt format	6	6	6	5.5
3) Is the way in which the questions (in the PPIQ) were presented confusing to you?	Overt format	5.5	6	5	5.5
4) Please rate the amount of time taken to complete this questionnaire.	Ease of use	5	5	4	5.5
5) To what extent do you think this questionnaire examines public involvement in decision making?	Purpose and framework	6	6	5	6.5
6) How many of the items are crucial or necessary, and how many are redundant or unnecessary?	Content validity	6	6	5	7
7) Do you think that there are important areas (gaps) that should be included in a measure of successful public involvement that have not been included?	Content validity	6	6	6	6
8) Do you think the response scale provided in the questionnaire allows you enough choice for your responses?	Ease of use	6	6.5	6	7
9) Do you think that it would be acceptable to others from the standpoint of understanding the questions, the time to complete and their acceptance of its comprehensiveness?	Ease of use	6	6.5	5	5
*One public member is included in the overall analysis. Three participants did not report their profession, which are included in the overall analysis.

2.2 Sensibility interviews. Of the 21 participants who completed the sensibility questionnaire, 14 agreed to a follow-up sensibility interview (three academic experts, four committee members from drug reimbursement committees in Canada, and seven patient group representatives). Interview data was coded according to Feinstein’s five criteria. During the analysis and coding an additional theme emerged: “clear terminology”.

Overall purpose and framework of the PPIQ. Participants thought the PPIQ addressed the overall purpose, to evaluate patient and public involvement on committees was appropriate as one participant stated: “I felt that the survey was actually pretty well put together” (INT 7).While most participants thought the PPIQ addressed the overall purpose, many wanted the questionnaire to identify the purpose up front in order for the participant to understand the reason for completing the survey: “Knowing the purpose of why and then circling back and showing them the outcomes, I think those are the most important pieces.” (INT 8).

Overt format. Participants highlighted the need to include a progress bar, which identifies how far along one is in the survey: “I find that useful. It’s just a bit of telling people that you’re making progress” (INT 4). Another participant suggested that the PPIQ have a start stop option to aid in the ease of its use: “Some people just for various reasons might not be able to sit down and do it within 20 minutes. So I would absolutely have the option that you can stop and start” (INT 11).

Face validity. Most participants thought that the PPIQ was clear. As one participant said, “I think the questions are … [written] in very easy to understand language. And I think the way they’re written, they would resonate with people” (INT 10). Another participant stated, “I don’t recall any particularly confusing [items]” (INT 5). Participants reported that the content in the PPIQ was comprehensive:

“I thought that it was really thorough. And I thought that it did a really good job asking about people’s thoughts on the public and patient involvement process from many different perspectives” (INT 3).

While participants agreed the PPIQ was comprehensive, they noted that this resulted in a lengthy questionnaire.

Content validity. Participants did not identify any missing items however, a few identified some redundant items. There was discussion around “whether public and patient, consistently need to be pulled apart” (INT 3) across the items. Another participant thought “the questions about the chair of the committee ensuring that these perspectives are considered. It’s a little bit redundant … do we really care that it’s the chair that makes it happen or it’s just part of the process?” (INT 5). Finally, one participant “wondered if maybe it [some items] could be consolidated” (INT 8). Participants also discussed reducing the number of response options.

Ease of use. Participants indicated that overall the PPIQ was easy to use. As one participant noted, “I think it flowed well. Like it didn’t feel like it was leading me anywhere. Which is important, right? It was clear and logical” (INT 8). A concern that they raised was the literacy level of the questionnaire: “I mean I think that anybody with below a high school education would have trouble with this questionnaire” (INT 4). However, another participant said “the language is also reflective of or applicable to who might be filling it out” (INT 11). One participant highlighted that each question was needed: “I understand you’d like to … shorten it a little bit. But truthfully, there's probably not a lot I would eliminate. You know, even just scanning over it again, I mean I think they all ask different things” (INT 2).

Clear terminology. Some participants thought that further clarification of terms and definitions used within the survey was required. Participants highlighted the importance of providing a clear definition for the term, “industry”. For example, “And the other one too just in terms of language, you referenced industry in this as well. I mean I interpret industry … as … pharma. That needs to be spelled out or just defined maybe at the outset” (INT 11). Another participant said that “I remember feeling that sometimes it was difficult to sort of differentiate … [between] patient versus public” (INT 5).

Based on the results of the sensibility testing combined with research team feedback, we revised the PPIQ (Version 4) prior to pilot testing. PPIQ (Version 4) was informed by the Phase 2 sensibility testing, prompting us to improve the face validity and the purpose and context of the PPIQ through more clearly defining its purpose in the introduction of the questionnaire. Also, we more clearly articulated the distinction between public and patient stakeholders throughout the questionnaire. To improve ease of use, we reduced the response options on each question from seven to five options, and because the PPIQ is designed to be administered electronically, we included a status bar to illustrate progress to improve user experience and the perceived length of the PPIQ.

Phase 3: Pilot Testing

A total of 55 committee members across Canada were contacted by email to complete the PPIQ (Version 4) of which 15 (27%) opened the PPIQ, and 14 (25%) participants completed the PPIQ. The sample consisted of professional drug committee members: physician (n = 7), pharmacist (n = 4) and academic/researcher (n = 3). The average age of participants was 52.4 years (SD ± 12.3 years), with eight (57%) participants identifying as male. The average time for participants to complete the PPIQ was 19:00 minutes (SD ± 13:46). Table 5 summarizes the pilot data within each of the nine earlier established evaluation criteria (Rosenberg-Yunger & Bayoumi, 2014). For seven of the nine evaluation criteria, the majority of participants (> 50%) agreed (inclusive of ‘agree’ and ‘strongly agree’) that their respective committee satisfied the criteria based on the items in the PPIQ (Table 5). Approximately 10% of all participants’ responses were ‘not applicable’ for PPIQ items. Three items were identified as having over one third of the total sample answering ‘not applicable’: Question 18, “Patient member(s) adequately represent patient perspectives during our committee deliberation” (71%, n = 10, ‘not applicable’); Question 19, “Public member(s) adequately represent public perspectives during committee deliberation” (43%, n = 6, ‘not applicable’), and Q53, “Our patient group submission guidelines are easy to find on our website” (43%, n = 6, ‘not applicable’).

Table 5

Pilot Responses (n = 14)
Criteria	Participant Responses
	Agree *		Disagree *		NA		Neutral
	Responses	%	Responses	%	Responses	%	Responses	%
Adequate Opportunity for Participation	14	100.0	0	0.00	0	0.00	0	0.00
Adequate Representation of Stakeholders	76	49.4	36	23.4	20	13.0	22	14.3
Fair Decision-Making Processes	46	65.7	11	15.7	4	5.7	9	12.9
Legitimacy of Committee Process	158	63.2	54	21.6	14	5.60	24	9.6
Meaningful Degree of Participation	79	62.7	18	14.3	20	15.9	9	7.1
Noticeable Effect on Decisions	45	40.5	35	31.5	17	15.3	14	12.6
Rationale and Roles of Patient	41	97.6	0	0.00	0	0.00	1	2.4
Sufficient support	23	82.1	1	3.6	0	0.00	4	14.3
Considerations of Efficiency	37	88.1	0	0.00	2	4.8	3	7.1
Total Responses	519	62.0	155	18.5	77	9.2	86	10.3
*Based on limited sample size, ‘Agree’ responses are inclusive of ‘Agree’, ‘Strongly Agree’ and ‘Disagree’ responses are inclusive of ‘Disagree’, ‘Strongly Disagree’.

Phase 1: Item generation and reduction

1.1

Item Bank. The preliminary item bank had a total of 846 items. This was refined internally resulting in an initial draft of the PPIQ that had 85 items. After reviewing the 85 items with the larger team, we reduced the PPIQ to 71 items, which was then organized according to the nine pre-specified criteria (Table 1) and presented to stakeholders for feedback.

We then organized the item bank putting similar items together (i.e., patient and public involvement, patient group submissions, role of the chair, etc.). We created brief instructions which outlined the purpose of the PPIQ, and a Likert scale with seven response options labelled from very strongly agree (0) to very strongly disagree (7). This resulted in an initial draft of the PPIQ (Version 1) to be presented to users for feedback.

1.2 User Feedback (Focus Group). Three people participated in the first focus group session (a committee member, patient group representatives, and a knowledge user partner representative). Participants commented on clarity of questions and instructions, appropriateness of response options, and missing questions. Participants discussed the issue of clarity as related to both the instructions of the PPIQ and specific items. Participants identified a need to better distinguish between the use of clinical and patient evidence. Others suggested the inclusion of a not applicable. Feedback from this focus group was reviewed by the research team and knowledge user, which resulted in PPIQ (Version 2). Many redundant items were removed from the PPIQ resulting in a reduction from 84 items to 72 items, and four open ended questions which were not present in the initial draft (Table 3) were added.

Table 3

Open ended questions suggested by focus group participants
Question 1	What other general comments do you have about patient and public involvement in the drug recommendation process?
Question 2	Think back to a time when patient input was not valued by the committee during the drug recommendation process, what was the reason for this?
Question 3	Think back to a time when patient input was not necessary in the drug recommendation process, why was this so?
Question 4	Can you think of a time when public and patient input was necessary in the drug recommendation process but did not add any value to the deliberation? Why was this so?

1.3 User Feedback (Online Survey). Given the low attendance during the first stakeholder feedback session via the focus group, we conducted the online user feedback data collection to gain additional feedback on the revised PPIQ (Version 2). A group of 14 individuals were contacted for the second stakeholder feedback session. Four participants (a committee member, a patient group representative, an industry employee, and a knowledge user partner representative) responded to the online survey. Participants identified missing questions including items around compensation and “adequacy of training on HTA [health technology assessment], including the clinical and economic reviews” (P2). Results from the online survey and electronic feedback were reviewed by the research team and knowledge user (CADTH) resulting in a revised PPIQ (Version 3).

In this study, we used a multi-methods multi-phased approach to develop and refine a questionnaire to evaluate patient and public involvement in committees making recommendations regarding public funding of drugs. The purpose of the PPIQ is to evaluate the level of ‘successful’ patient and public involvement based on nine criteria allowing for a dynamic evaluation of public and patient involvement across different drug recommendation committee contexts (10). By eliciting stakeholder feedback, we were able to ensure that the PPIQ had face and content validity and that it met the expectations of potential users, including patient group representatives, industry, and patient and public committee members. Based on participants’ responses to the sensibility questionnaire and on interview data, the content was appropriate, and the items were necessary and contributed to adequately evaluating public and patient involvement on drug reimbursement committees. Our pilot indicates the feasibility of delivering the PPIQ electronically (based on response rate and time to complete), and demonstrates the applicability of questions across multiple committees. While the sample of the pilot was small, the method for summarizing items within criteria may be useful for future implementations, such as testing reliability or construct validity of the PPIQ with drug funding committees in Canada.

The sensibility interview results indicate that the practicality of implementing the PPIQ among different committees may vary based on the number and type of members, as well as available time and resources and competing committee priorities. Overall participants involved in the sensibility testing reported that the PPIQ was easy to use; however, this criterion had the lowest median score, specifically related to its length (median score of 5/7). The sensibility interview data suggested the length may be a limitation when implementing the PPIQ in the ‘real world’. These pilot data suggested that on average it took participants approximately 20 minutes to complete the PPIQ, which may or may not be appropriate for committee members with competing priorities. Nonetheless, participants acknowledged the complexity of patient and public involvement in drug reimbursement committees, and in turn understood the need for the PPIQ to be robust and comprehensive.

The PPIQ is one of few questionnaires to measure public and patient involvement within formulary recommendations for public funding. Another questionnaire that may be used to evaluate public and patient involvement in our context is the Public and Patient Engagement Evaluation questionnaire (PPEET) (18). The PPEET was initially developed in 2011 to measure engagement broadly, allowing the questionnaire to be used in a variety of contexts in Canada with the objective of improving the quality of public and patient engagement. In contrast, the PPIQ focuses specifically on drug reimbursement committees, and as a result provides an evaluation of member involvement that is specific to the context drug committees in Canada.

Some limitations are present in our approach. Throughout our user testing (phase 1.2, 1.3) we had low participant numbers; however, this phase was also informed by the key informant interviews and literature review conducted previously by AMB and ZRY. In our sensibility testing, we had only one public member and no patient committee members, but we sought to generate patient and public perspectives through representation from multiple patient group members. When completing the pilot test of the PPIQ there was only representation from professional committee members (i.e., physicians, pharmacists, researchers, etc.), which highlights the need for further testing of the PPIQ with public and patient committee members. Despite these limitations, the PPIQ was developed across multiple phases, with opportunity for a breadth of perspectives from across stakeholder groups.

The methods used for the development of the PPIQ, including qualitative interviews, literature review, user feedback, sensibility and pilot testing, have resulted in a questionnaire that may be used with committees making drug funding recommendations in Canada. This questionnaire may have applicability to other geographic contexts with similar priority setting healthcare structures (5).

This research was approved by Research Ethics Boards at Ryerson University (Toronto, Ontario, Canada [REB# 2015–079]) and University of Toronto (Toronto, Ontario, Canada [REB #30921]). All participants provided their consent to participate as well as their consent for publication. The authors have no competing interests or conflicts of interest to declare. This study was funded by CIHR (Operating Grant 136887). All authors contributed to the conception of this funded research project, in addition to writing and reviewing this manuscript. Kelly K. O’Brien is supported by a Canada Research Chair (Tier 2) in Episodic Disability and Rehabilitation. Ahmed Bayoumi was supported by the Fondation Baxter and Alma Ricard Chair in Inner City Health at St. Michael’s Hospital and the University of Toronto. The datasets generated during and/or analysed during the current study are not publicly available. Due to the limited number of drug reimbursement committee members in Canada, the data will be kept private to retain the confidentiality of research participants. We acknowledge and thank all research participants.

Box 1. Sensibility Questionnaire

Please answer the following questions by circling one option below.

1. Please rate the questionnaire in terms of clarity and simplicity:

1234567

Unacceptable Poor GoodExcellent

Indicate which questions were not clear:

2. Were the questionnaire instructions adequate?

1234567

Unacceptable Poor GoodExcellent

3. Is the way in which the questions were presented confusing to you?

1234567

Extremely Very Moderately Not at all

Indicate any questions that were confusing:

4. Please rate the amount of time taken to complete this questionnaire:

1234567

Unacceptable Needs Work Acceptable Perfect

5. To what extent do you think this questionnaire examines public involvement in decision making?

1234567

Unacceptable Needs Work Acceptable Perfect

6. How many of the items are crucial or necessary, and how many are redundant or unnecessary?

1234567

Many Unnecessary Some Unnecessary Few Unnecessary None

Indicate the items that were redundant or unnecessary:

7. Do you think that there are important areas (gaps) that should be included in a measure of successful public involvement that have not been included?

1234567

Crucial Gaps Important Gaps Minor Gaps Minimal Gaps

Indicate areas that you think should be included:

8. Each question in the questionnaire has a seven -point response scale. Do you think the response scale provided in the questionnaire allows you enough choice for your responses?

1234567

Unacceptable Poor Good Excellent

9. Having completed the questionnaire do you think that it would be acceptable to others from the standpoint of understanding the questions, the time to complete and their acceptance of its comprehensiveness?

1234567

Unacceptable Poor Good Excellent

Box 2. Sensibility Testing Interview Guide

Thank you for agreeing to participate in this interview. Before we begin did you have any questions about the consent form?

After participants have completed the sensibility questionnaire we will go through the PPIQ and ask following questions:

1. Do you think the title of the questionnaire is appropriate?

2. Do you think the items address the questionnaire’s purpose to evaluate patient and public involvement on committees?

Face and Content Validity

1. What are your overall thoughts on the questionnaire items? [Probes: How well do you think the items captured the public & patient involvement experience?]

2. Do you think any items particularly captured your experience with patient and public involvement? [Probes: If so, please identify the questions?]

3. Do you think the response options were sufficient to allow you to adequately answer the question to best describe your experience? [Probes: If yes, what did you like about them? If no, what would you change?

Item Generation

4. Do you think there were any items missing from the questionnaire? [Probes: If yes, what types of questions would you add?]

Item Wording

5. What are your thoughts about the wording of the questions in the questionnaire?

Item Reduction

6. Do you think there were any questions that were redundant or repetitive? [Probes:

If yes, please identify those questions?]

Ease of Usage

7. What did you think of the length of time it took you to complete the questionnaire? [Probes:

Too long?]

Conclusion

Looking back on the instrument please comment on the instrument as a whole [probes: flow and context effects]. Do you have anything else you wish to add about the draft instrument that you completed?

Thank you for participating!

Boote J, Telford R, Cooper C. Consumer involvement in health research: a review and research agenda. Health Policy [Internet]. 2002 Aug [cited 2019 Jan 10];61(2):213–36. Available from: .
Conklin A, Morris Z, Nolte E. What is the evidence base for public involvement in health-care policy?: results of a systematic scoping review. Heal Expect [Internet]. 2015 Apr [cited 2019 Jan 10];18(2):153–65. Available from: .
Gauvin F-P, Abelson J, Giacomini M, Eyles J, Lavis JN. Moving cautiously: Public involvement and the health technology assessment community. Int J Technol Assess Health Care [Internet]. 2011 Jan 25 [cited 2019 Jan 10];27(1):43–9. Available from: .
Pivik J, Rode E, Ward C. A consumer involvement model for health technology assessment in Canada. Health Policy (New York) [Internet]. 2004 Aug [cited 2019 Jan 10];69(2):253–68. Available from: .
Rosenberg-Yunger ZRS, Daar AS, Thorsteinsdóttir H, Martin DK. Priority setting for orphan drugs: An international comparison. Health Policy (New York). 2011.
University of Toronto Priority Setting in Health Care Research Group
Bruni RA, Laupacis A, Martin DK. University of Toronto Priority Setting in Health Care Research Group. Public engagement in setting priorities in health care. CMAJ [Internet]. 2008 Jul 1 [cited 2019 Jan 10];179(1):15–8. Available from: .
Rosenberg-Yunger ZRS, Thorsteinsdóttir H, Daar AS, Martin DK. Stakeholder involvement in expensive drug recommendation decisions: An international perspective. Health Policy (New York) [Internet]. 2012 May 1 [cited 2019 Jan 10];105(2–3):226–35. Available from: .
10.1186/s40900-016-0036-9
Berglas S, Jutai L, MacKean G, Weeks L. Patients’ perspectives can be integrated in health technology assessments: an exploratory analysis of CADTH Common Drug Review. Res Involv Engagem [Internet]. 2016 Dec 7 [cited 2019 Jan 10];2(1):21. Available from: .
Dipankui MT, Gagnon M-P, Desmartis M, Légaré F, Piron F, Gagnon J, et al. Evaluation of patient involvement in a health technology assessment. Int J Technol Assess Health Care [Internet]. 2015 Jun 11 [cited 2019 Jan 10];31(3):166–70. Available from: .
Rosenberg-Yunger ZRS, Bayoumi AM. Evaluation criteria of patient and public involvement in resource allocation decisions: a literature review and qualitative study. Int J Technol Assess Health Care [Internet]. 2017 May 18 [cited 2019 Jan 10];33(2):270–8. Available from: .
Feinstein AR. The Theory and Evaluation of Sensibility. In: Clinimetrics. 2018.
Feinstein AR. The Intellectual Crisis in Clinical Science: Medaled Models and Muddled Mettle. Perspect Biol Med [Internet]. 1987 [cited 2019 Jan 10];30(2):215–30. Available from: .
O’Brien KK, Bayoumi AM, Bereket T, Swinton M, Alexander R, King K, et al. Sensibility assessment of the HIV Disability Questionnaire. Disabil Rehabil. 2013.
Rowe BH, Oxman AD. An assessment of the sensibility of a quality-of-life instrument. Am J Emerg Med. 1993.
Team RC. R version 3.2.0 [Internet]. 2019 [cited 2020 Mar 10]. Available from: .
Denzin NK, Lincoln YS. Introduction. The Discipline and Practice of Qualitative Research. The Sage handbook of qualitative research. 3rd ed. Thousand Oaks: Sage Publications Ltd; 2005. pp. 1–32.
QSR. NVivo 10 research software for analysis and insight. 2014.
Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: Development and usability testing of the Public and Patient Engagement Evaluation Tool. Heal Expect. 2016.

Download PDF

Version 1

posted

You are reading this latest preprint version

Development of the Patient and Public Involvement Questionnaire (PPIQ) for Canadian Drug Funding Committees

Status:

Version 1

Abstract

Figures

Background

Methods

Phase 3: Pilot Testing

Results

Phase 1: Item generation and reduction

Discussion

Conclusions

Declarations

References

Status:

Version 1