Improving the patient-reported outcome sections of clinical trial protocols: a mixed methods evaluation of educational workshops

doi:10.21203/rs.3.rs-720930/v1

Introduction

Failure to incorporate key patient-reported outcome (PRO) content in trial protocols affects the quality and interpretability of the collected data, contributing to research waste. Our group developed evidence-based training specifically addressing PRO components of protocols. We aimed to assess whether 2-day educational workshops improved the PRO completeness of protocols against consensus-based minimum standards provided in the SPIRIT-PRO Extension in 2018.

Method

Annual workshops were conducted 2011–2017. Participants were investigators/ trialists from cancer clinical trials groups. Although developed pre-2018, workshops covered 15/16 SPIRIT-PRO items. Participant feedback immediately post-workshop and, retrospectively, in November 2017 was summarised descriptively. Protocols were evaluated against SPIRIT-PRO by two independent raters for workshop protocols (developed post-workshop by participants) and control protocols (contemporaneous non-workshop protocols). SPIRIT-PRO items were assessed for completeness (0 = not addressed, 10 = fully addressed). Mann-Whitney U-tests assessed whether workshop protocols scored higher than controls by item and overall.

Results

Participants (n = 107) evaluated the workshop positively. In 2017, 16/41 survey responders (39%) reported never applying in practice; barriers included role restrictions (14/41, 34%) and lack of time (5/41, 12%). SPIRIT-PRO overall scores did not differ between workshop (n = 16, mean = 3.84 (standard deviation = 1.86)) and control protocols (n = 6, 2.46 (1.35)), (p = 0.09). Workshop protocols scored higher than controls on two items: ‘specify PRO concepts/domains’ (p = 0.035); ‘specify analysis methods’ (p = 0.011).

Conclusion

Although participants were highly satisfied with these workshops, the completeness of PRO protocol content improved only slightly. Additional knowledge translation efforts are needed to assist protocol writers address SPIRIT-PRO guidance and avoid research waste that may eventuate from sub-optimal PRO protocol content.

Psychology

Educational Psychology

Cancer Biology

SPIRIT-PRO

PRO protocol

ANZCTR

physical

emotional

social

The protocol of a clinical trial is the foundation for study planning, conduct, reporting, and appraisal(1). It should therefore provide sufficient detail to facilitate these purposes in order to avoid wasting research resources on poorly planned, implemented and reported trials(2, 3). These principles apply to all trial endpoints, including patient-reported outcomes (PROs), which complement clinical outcomes by providing patients’ perceptions of the impact of disease and treatment. PROs include symptoms and various aspects of function (e.g. physical, emotional, social) and multi-dimensional constructs such as health-related quality of life. The importance of PROs is widely acknowledged, and PROs are commonly included in clinical trials. For example, 27% of 96,736 trials registered in ClinicalTrials.gov between 2007 and 2013(4) and 45% of 13,666 trials registered in the Australia New Zealand Clinical Trials Registry (ANZCTR) between 2005 to 2017(5) included one or more PROs.

Two key strategies for achieving high quality data in clinical trials are to standardise methods for endpoint assessment across patients and sites and to minimize missing endpoint data. This is particularly pertinent for PROs as they are subjective phenomena that are typically assessed repeatedly over time and cannot be retrieved retrospectively. It is therefore worrying that research nurses working in clinical trials report receiving insufficient information in trial protocols to implement PRO data collection consistently(6, 7), with some saying they had to revert to training received for previous trials in the absence of specific protocol instructions. Further, some noted that planned PRO assessments were often missed because the protocol failed to provide contingency plans for capturing PRO data when participants missed a clinical appointment or scheduled assessment(6). Extensive rates of avoidable missing PRO data reduces the study sample size(8, 9), risk biased and unreliable trial results, and can led to PRO endpoints not being reported(10, 11). This in turn may lessen the impact of PROs on routine clinical care, mislead clinical or health policy decision-making, reduce the value of patient participation in trials and waste limited healthcare and research resources(2, 3). This calls into question the ethics of collecting PRO data that will not be used(12).

Systematic reviews have shown that trial protocols often lack important information with regard to PROs(13–15). This is concerning for various reasons. For example, if a protocol fails to justify the purpose of PRO assessment, or if the coverage of the PRO endpoint is poor, the trial staff may interpret that PROs are not as valued as other trial endpoints and invest less time and effort into ensuring high quality data collection. If there is no clear PRO research question or hypothesis, or if rates of missing PRO data are high, there may be little incentive to analyse the PRO data or report it in a meaningful way – again risking research waste. Indeed, there is evidence that poor PRO coverage in protocols is correlated with poor reporting of PRO results(11, 14). To address these issues, the SPIRIT-PRO guidance was released in 2018, providing consensus-based guidance to facilitate international best practice standards for minimum PRO content in clinical trial protocols(16).

The Sydney Quality of Life Office (SQOLO) was funded by the Australian Government through Cancer Australia from 2011–2021 to support the national network of cancer clinical trials groups (CCTG) to include PRO endpoints in their trials/studies to international best standards. A core activity of the SQOLO was providing workshops to educate CCTG members on the scientific and logistic considerations for designing a PRO study and how these aspects should be addressed in the clinical trial protocol. A series of 2-day educational workshops was run annually 2011–2017. Although the SPIRIT-PRO guidance had not been developed at that time, members of our team (MK, RMB) were also members of the executive group that led the development of the SPIRIT-PRO guidance. In 2011, we (MK, RMB) developed the PROtocol Checklist which underpinned the content of our workshops. The PROtocol Checklist was the primary pre-cursor to the SPIRIT-PRO; it covered 13 of the 16 SPIRIT-PRO items plus several more. Workshop content covered 15 of the 16 SPIRIT-PRO checklist items.

The aim of this study was to assess whether this 2-day educational workshop directed at oncology trialists and clinician researchers from CCTGs was effective in fostering the inclusion of SPIRIT-PRO items in trial protocols. Three specific research questions were addressed: 1) Were participants satisfied with the workshop content and format?; 2) Did participants use the PROtocol Checklist in the long-term?; 3) Were the PRO components of protocols brought to the workshop more complete than those of contemporaneous protocols not brought to the workshop?

The PROtocol Checklist Workshop

SQOLO staff (MK, RMB, CR plus two statisticians) conducted 2-day face-to-face workshops annually between 2011–2017 to educate investigators and trialists about key aspects of PRO assessment within clinical trials and how to address these in protocols. Workshop format, resources and and topics covered each year are outlines in Box 1 and online supplement 1. Each of the 14 Australian CCTGs was invited to nominate one or more members to attend a workshop each year, and each attendee was encouraged to bring a protocol-in-development for further development.

Prior to developing the first workshop, we developed a PROtocol Checklist to specify how PROs should be addressed in each section of a trial protocol, drawing on guidance documents and research of three key international trials groups that pioneered excellence in HRQL research: the Canadian Cancer Trials Group (CCTG), the European Organisation for Research and Treatment of Cancer (EORTC) and SWOG (formerly Southwest Oncology Group). The first version of the checklist was presented in the 2011 workshop. It was updated twice to improve formatting and refine items. The third and final version (online supplement 2) was used in workshops 2013–2017. The PROtocol Checklist was available on the SQOLO website.

Aim 1 – Post-workshop evaluation survey (immediate)

All participants were invited to anonymously evaluate the workshop at the end of Day 2. The survey consisted of 11 questions (online supplement 3) which assessed: usefulness of the workshop and PROtocol Checklist resource; what the most valuable topic covered was; what aspects could be improved; whether the real protocol examples provided were useful; and who from the CCTGs would benefit most from future workshops. Finally, participants were asked to provide an overall rating of the workshop on a scale of 1 (poor) to 10 (excellent).

Aim 2 – Long-term research practice survey

In November 2017 everyone who had attended the workshops held between 2011 and 2017 were invited to complete an online survey via REDCap (online supplement 4). Respondents were asked whether and how they had used the PROtocol Checklist since they attended the workshop, whether they anticipated using it in the future, and barriers to its use.

Aim 3 – Completeness of PRO content of protocols

To assess the extent to which protocols brought to/developed during the PROtocol Checklist Workshop addressed recommended items for inclusion in PRO sections of trial protocols, as compared to a control sample of study protocols, we contacted key members of the CCTGs and past workshop participants to identify and request eligible protocols. Workshop protocols were eligible for inclusion if they were brought to the PROtocol Checklist Workshops between 2011 and 2017; were led by one of the Australian cancer clinical trial groups; included a PRO endpoint; were finalised for study activation; and permission was obtained from the trial group or the trial principal investigator to include results of the protocol’s review in analyses for this paper. Eligibility criteria for control protocols were identical, with the exception that control protocols were only eligible for inclusion if they were not brought to the PROtocol Checklist Workshops between 2011 and 2017 and did not receive input on development from any SQOLO staff.

Each protocol was evaluated against the full complement of items covered by the SPIRIT-PRO Checklist (online supplement 5) and the PROtocol Checklist by two independent trained raters (either FM or RC and either CB, SC, DL, JS). Raters were blinded to workshop/control status of each protocol. To ensure a standardised rating process, two experienced raters (FM and RC) developed a comprehensive rating guide for each checklist item (online supplement 6). All items were rated for completeness on a scale from 0 (not addressed) to 10 (fully addressed). Ratings discrepant by up to two points were averaged. Discrepancies of three or more points were discussed and resolved with MK.

Analysis

Participant survey data. Workshop participants’ survey data was summarized using descriptive statistics; percentages for questions with categorical responses and means for questions with numerical responses. Free-text responses were analysed using content analysis by MT and RC.

Protocol evaluations. The primary analysis related to SPIRIT-PRO checklist; PROtocol checklist items not covered by SPIRIT-PRO were addressed as a secondary analysis. An overall SPIRIT-PRO completeness score was calculated by computing the average of all 16 SPIRIT-PRO item scores. The mean (M) and standard deviation (SD) of completeness scores were then calculated for each SPIRIT-PRO Checklist item, the overall SPIRIT-PRO score, and each additional PROtocol Checklist item. Differences in the distribution of completeness scores between the control and workshop protocols were examined using Mann-Whitney U tests (SPSS V24). For graphical presentation, the 0 to 10 scale was categorised as follows: 0 (not addressed), 1–3 (poorly addressed), 4–6 (acceptably addressed), 7–9 (well addressed), 10 (fully addressed).

Participants and post-workshop evaluation survey

From 2011 to 2017, 107 people participated in PROtocol Checklist Workshops. Participants were spread relatively evenly across years and trial groups (Table 1). All but one participant completed post-workshop evaluation surveys. All 106 respondents stated that the workshop either met or exceeded their expectations, all found the examples of real protocols shown during the workshop useful (2013–2017), and all indicated that the PROtocol Checklist resource was useful with nearly a quarter stating that they planned to use it in the future. Sessions commonly noted as most useful were: PRO measures, PRO questionnaire administration, and Missing Data - Statistics and Logistics. About a third of respondents stated that the sessions on statistical considerations for PROs were difficult to follow for people with no background in statistics. Some participants found the session on Utility Measures too complex and not relevant to them. Areas noted for improvement were: more time needed for group discussions; provide reading materials before the workshop to help participants prepare.

Table 1

Characteristics of workshop participants
	Workshop Attendees N = 107	Invited to Follow-up survey (valid email) n = 82	Follow-up Survey n = 41	Time since w/s (years)	Follow-up Survey Non-responders N = 41
Year attended Workshop 2011 2012 2013 2014 2015 2016 2017	14 14 19 13 16 15 16	8 10 15 11 13 11 14	2 4 7 7 5 7 9	7 6 5 4 3 2 1	6 6 8 3 8 4 5
CTG AGITG ALLG ALTG (now TOGA) ANZBCTG (now BCT) ANZCHOG ANZGOG ANZMTG (now MASC) ANZUP ASSG (now ANZSA) COGNO PC4 PoCoG TROG	6 8 9 7 4 6 16 6 6 8 11 9 11	5 7 9 4 4 3 10 6 4 6 8 8 8	1 5 4 4 2 0 2 2 2 4 5 4 5	-	4 2 5 0 2 3 8 4 2 2 3 4 3
AGITG Australasian Gastro-Intestinal Trials Group; ALLG Australasian Leukaemia and Lymphoma Group; ALTG Australasian Lung Cancer Trials Group; ANZBCTG Australian New Zealand Breast Cancer Trials Group; ANZCHOG Australian and New Zealand Children’s Haematology and Oncology Group; ANZGOG Australia New Zealand Gynaecological Oncology Group; ANZMTG Australia and New Zealand Melanoma Trials Group; ANZSA Australia and New Zealand Sarcoma Association; ANZUP Australian and New Zealand Urogenital and Prostate Trials Group; ASSG Australasian Sarcoma Study Group; BCT Breast Cancer Trials; COGNO Cooperative Trials Group for Neuro-Oncology; MASC Melanoma and Skin Cancer Trails Limited; PC4 Primary Care Collaborative Cancer Clinical Trials Group; PoCoG Psycho-Oncology Co-operative Research Group; TOGA Thoracic Oncology Group Australasia; TROG Trans-Tasman Radiation Oncology Group

The individuals perceived as being most likely to benefit from attending the workshop were protocol developers, trial coordinators, project officers, central operations staff, principal investigators, co-investigators and research fellows. The overall rating of the workshop by participants (2013–2017, n = 80) ranged from 6 to 10 (where 1 = poor and 10 = excellent), with a median of 9, mean of 8.6 and SD of 0.96.

Written comments from respondents (Box 2) supported the results above .

Follow-up survey

In 2017, we attempted to contact all 107 workshop participants to complete the online follow-up survey. Many had changed workplaces and of these, we were unable to locate new contact details for 25. Of the 82 participants who were successfully contacted, 41 completed the survey. Respondents were trial coordinators (4/41, 10%), principal investigators (8/41, 20%), co-investigators (8/41, 13%), CCTG protocol developers (4/41, 10%), or their roles were not specified (17/41, 41%). Based on survey responses, since attending the workshop, 23/41 (56%) of respondents reported using the checklist when developing new protocols and 10/41 (24%) when amending protocols, while 16/41 (39%) said they had not used the checklist. The majority of respondents 32/41 (78%) indicated that the checklist would be useful if they were to develop a new study that included PROs.

Additional qualitative data provided as comments fell into two themes: how participants used the PROtocol Checklist since the workshop and barriers to using the PROtocol Checklist. Many of the comments supported the quantitative result noted above: over half (23/41, 56%) reported using the PROtocol Checklist when developing PRO endpoints for new trial protocols, and almost a quarter noted using it to review PRO components of existing protocols (10/41, 24%). Other uses included: guide for what should be included in the protocol (1/41, 2.4%); integration of checklist items into the CCTG standard protocol template to improve the PRO aspects of the protocol. (1/41, 2.4%); as a reference in grant applications (2/41, 5%); and as a training resource for research assistants (1/41, 2.4%).

Regarding barriers to using the PROtocol Checklist, one in five participants (8/41, 20%) noted not having an opportunity to use the checklist, due to either leaving their role, it not being relevant to their role, or it not being applicable to the trials they worked with. Other barriers included: time constraints within the busy protocol development process (4/41, 10%); forgetting to use the Checklist (3/41, 7%); finding it hard to use and the wording of some items hard to follow (2/41, 5%); not being able to find the Checklist when needed (1/41, 2.4%); protocol developers/investigators not seeing the value of adding additional detail on PROs (1/41, 2.4%); investigators wanting the protocol to be brief (1/41, 2.4%).

Protocol Evaluation

Of the 74 protocols brought to the PROtocol Checklist Workshops, 25 were finalised to study activation, of which only 16 were found to be eligible for review. In total 22 finalised protocols were obtained from the Australian CCTGs and assessed against the PROtocol Checklist. Table 2 summarises the characteristics of the control protocols (n = 6) and the workshop protocols (n = 16).

Table 2

Characteristics of the control and workshop protocols
	Control protocol n = 6	Workshop protocol n = 16
Date of protocol version 2010 2011 2012 2013 2014 2015 2016 2017	0 (0%) 1 (16.7%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (16.7%) 4 (66.7%)	2 (12.5%) 0 (0%) 0 (%) 2 (12.5%) 2 (12.5%) 3 (18.8%) 6 (37.5%) 1 (6.3%)
PRO Endpoint(s) Primary only Secondary only Primary & Secondary Exploratory	2 (33.3%) 3 (50%) 1 (16.7%) 0 (0%)	1 (6.3%) 12 (75.0%) 2 (12.5%) 1 (6.3%)
Trial Phase Phase I (biomed-safety) Phase 1 (screening-feasibility) Phase II (biomed) Phase III (biomed) Correlative study	0 (0%) 3 (50%) 2 (33.3%) 0 (0%) 1 (16.7%)	0 (0%) 0 (0%) 9 (56.2%) 7 (43.8%) 0 (0%)
Cancer type Brain Breast Gastro-intestinal Leukaemia Lung Melanoma Prostate Testicular Two or more cancer types	2 (33.3%) 0 (0%) 2 (33.3%) 0 (0%) 0 (0%) 1 (16.7%) 0 (0%) 0 (0%) 1 (16.7%)	1 (6.2%) 1 (6.2%) 0 (0%) 3 (18.8%) 3 (18.8%) 4 (25.0%) 1 (6.2%) 1 (6.2%) 2 (12.6%)
Cancer Stage Pre cancer (at risk) Early solid tumours Advanced solid tumours Mixed stage solid tumours NA (blood cancers)	1 (16.7%) 1 (16.7%) 3 (50.%) 1 (16.7%) 0 (0%)	1 (6.2%) 5 (31.2%) 7 (43.8%) 0 (0%) 3 (18.8%)
Healthcare stage Screening Curative treatment Palliative treatment Psychosocial/supportive care Follow-up Mixed healthcare stages	2 (33.3%) 3 (50.0%) 0 (0%) 0 (0%) 0 (0%) 1 (16.7%)	1 (6.2%) 9 (56.3%) 1 (6.3%) 4 (25.0%) 1 (6.2%) 0 (0%)
Active & palliative treatment types Chemotherapy Chemotherapy or radiotherapy Immunotherapy Surgery only Surgery & chemotherapy Surgery & radiotherapy Targeted therapy NA (other healthcare stages)	0 (0%) 0 (0%) 1 (16.7%) 0 (0%) 0 (0%) 0 (0%) 2 (33.3%) 3 (50.0%)	3 (18.8%) 1 (6.2%) 1 (6.2%) 2 (12.6%) 0 (0%) 1 (6.2%) 2 (12.5%) 6 (37.5%)

Table 3 presents mean SPIRIT-PRO completeness scores for the control and workshop protocols, and the results of Mann-Whitney U tests for difference in distribution of scores between control and workshop protocols. The mean SPIRIT-PRO total score for workshop protocols was 3.84/10 (SD = 1.86) versus 2.46/10 (SD = 1.35) for control protocols; the distribution of this score did not differ significantly between workshop and control protocols (p = 0.09). Despite no significant difference based on overall scores, analysis of components within the checklist highlighted between group differences for a number of items. For the SPIRIT-12-PRO item: ‘Specify the concepts/ domains used to evaluate the intervention’, workshop protocols were more complete than control protocols (M_W = 4.45, M_C = 1.75, p = .035). Analysis of the three subcomponents of this item revealed that this difference were largely driven by the workshop protocols being more complete than control protocols (M_W = 4.50, M_C = 1.08, p = .021) on specifying the analysis metric for each PRO concept or domain used to evaluate the intervention. Workshop protocols were also more complete than control protocols (M_W = 5.23, M_C = 1.54, p = .011) on SPIRIT-20a-PRO: ‘PRO analysis methods’. Analysis of this item’s two subcomponents indicated this was largely due to workshop protocols being more complete than control protocols (M_W = 7.34, M_C = 3.08, p = .011) on stating PRO analysis methods. Of note, none of the protocols addressed SPIRIT-5a-PRO: ‘specifying the individual responsible for the PRO content’, or SPIRIT-20c-PRO: ‘state how missing data will be described’.

Table 3

Mean (M) and standard deviation (SD) of the completeness rating¹ for each SPIRIT-PRO item and the total SPIRIT-PRO score for control and workshop protocols
SPIRIT-PRO item	Control protocol M (SD)	Workshop protocol M (SD)	p value^2,*
SPIRIT-5a-PRO: Individual responsible	0.00 (0.00)	0.00 (0.00)	/
SPIRIT-6a-PRO: Rationale and research question	2.67 (2.41)	4.42 (3.35)	0.395
- Summarise PRO findings from past relevant studies	2.25 (2.99)	5.09 (3.83)	0.110
- Describe the rationale for PRO assessment	3.75 (3.31)	5.09 (3.64)	0.372
- Describe the PRO specific research question	2.00 (1.45)	3.06 (3.43)	0.822
SPIRIT-7-PRO: Objectives & hypotheses	4.17 (1.60)	5.41 (2.90)	0.251
SPIRIT-10-PRO: Eligibility criteria	4.00 (4.89)	6.25 (5.00)	0.356
- Specify any PRO-specific eligibility criteria	4.00 (4.90)	6.25 (5.00)	0.356
- If PROs will not be collected in the entire study sample, provide a rationale and describe the method for obtaining the PRO subsample. Note. N/A is an option	-	-	/
SPIRIT-12-PRO: Concepts & domains	1.75 (2.29)	4.45 (2.97)	0.035*
- Specify the PRO concepts/domains used to evaluate the intervention.	2.75 (2.36)	5.47(3.04)	0.076
- For each of the PRO concepts or domains used to evaluate the intervention, specify the analysis metric.	1.08 (1.96)	4.50 (3.46)	0.021*
- For each of the PRO concepts or domains used to evaluate the intervention, specify the principal time point or period of interest.	1.42 (2.80)	3.37 (3.44)	0.140
SPIRIT-13-PRO: Participant timeline	3.29 (1.25)	4.19 (1.70)	0.302
- Include a schedule of PRO assessments, specifying which measures will be used at each assessment.	6.75 (4.05)	8.81 (2.35)	0.171
- Provide a rationale for the assessment time points.	1.08 (1.53)	2.13 (2.46)	0.475
- Is initial PRO assessment pre-randomisation? If initial PRO assessment is post-randomisation provide a justification.	10.00 (0.0)	6.33 (4.80)	0.287
- Specify PRO assessment time windows.	3.42 (4.13)	4.13 (3.52)	0.503
- Specify whether PRO collection is prior to clinical assessments.	2.50 (5.00)	2.93 (4.65)	0.748
- If using multiple questionnaires, specify whether order of administration will be standardized. Note N/A is an option	0.00 (0.00)	0.00 (0.00)	/
SPIRIT-14-PRO : Sample size	2.25 (3.03)	4.50 (5.02)	0.386
SPIRIT-18a(i)-PRO : PRO instrument	2.35 (2.17)	3.55 (1.61)	0.161
- Justify the PRO instrument to be used.	3.33 (2.82)	4.94 (2.64)	0.237
- Describe the PRO instrument in terms of domains, number of items, recall period, instrument scaling/scoring.	3.33 (1.94)	4.56 (2.80)	0.459
- Evidence of PRO instrument measurement properties, interpretation guidelines, and patient acceptability/burden should be provided or cited if available, ideally in the population of interest.	1.58 (2.13)	2.53 (2.44)	0.353
- State whether the measure will be used in accordance with any user manual and specify and justify deviations if planned.	1.17 (2.86)	2.16 (3.59)	0.373
SPIRIT-18a(ii)-PRO: Mode & setting of administration	6.04 (4.42)	5.56 (3.94)	0.528
- Include a data collection plan outlining the permitted mode(s) of administration.	5.42 (4.72)	4.25 (4.11)	0.408
- Specify PRO data collection setting (e.g. clinic, home, other).	6.67 (5.16)	6.88 (4.79)	0.927
SPIRIT-18a(iii)-PRO: Translations	2.13 (2.42)	3.41 (3.73)	0.645
- Specify whether more than one language version will be used.	2.17 (2.38)	3.41 (3.73)	0.673
- If a translation will be used, state whether it was developed using currently recommended methods. Note. N/A is an option	-	-	/
SPIRIT-18a(iv)-PRO: Proxy assessment	All N/A	7.00 (1.41)³	/
SPIRIT-18b(i)-PRO: Strategies for minimising missing data	2.42 (2.15)	1.97(3.05)	0.407
SPIRIT-18b(ii)-PRO: Discontinuation or withdrawal	4.17 (2.64)	5.41 (3.18)	0.350
SPIRIT-20a-PRO: PRO analysis methods	1.54 (1.48)	5.23 (3.21)	0.011*
- State PRO analysis methods.	3.08 (2.96)	7.34 (2.71)	0.011*
- State any plans for addressing multiplicity or type 1 (α) error.	0.00 (0.00)	3.13 (4.79)	0.128
SPIRIT-20c-PRO: Missing data	0.13 (0.31)	1.05 (1.6)	0.215
- State how missing data will be described.	0.00 (0.00)	0.00 (0.00)	/
- Outline the methods for handling missing items and entire assessments (e.g. approach to imputation and sensitivity analyses).	0.25 (0.61)	2.09 (3.21)	0.215
SPIRIT-22-PRO: Harms	0.00 (0.00)	2.08 (3.82)	0.188
- State whether or not PRO data will be monitored during the study to inform the clinical care of individual trial participants.	0.00 (0.00)	2.50 (4.47)	0.186
- If PRO data will be monitored during the study to inform clinical care of individual participants, state how this will be managed in a standardized way. Note. N/A is an option.	-	7.50 (5.00)⁴	/
- If PRO data will be monitored during the study to inform clinical care of individual participants, describe how this process will be explained to participants e.g. in the participant information sheet and consent form. Note. N/A is an option.	-	7.50 (5.00)⁴	/
Total SPIRIT-PRO Score	2.46 (1.35)	3.84 (1.86)	0.098
1. Higher scores indicate more completely addressed items.
2. Mann-Whitney U tests, * indicates statistical significant at 95% level (p < 0.05)
3. Based on n = 2
4. Based on n = 4

Figure 1 shows completeness ratings of the SPIRIT-PRO items in the 16 protocols developed by workshop participants in terms of percentage not addressed (rated 0), poorly addressed (rated 1–3), acceptably addressed (rated 4–6), well addressed (rated 7–9), and fully addressed (rated 10). Most items were predominantly poorly addressed and very few items were fully addressed. Only two items were at least 50% acceptably or well-addressed.

Table 4 presents the mean completeness scores on the 18 additional PROtocol Checklist items for the control and workshop protocols, and the Mann-Whitney U test results. The workshop protocols were more complete than the control protocols ((M_W = 3.94, M_C = 0.58, p = .011) on describing methods for deriving PRO endpoints from PRO data. However, control protocols were more complete than the workshop protocols (M_C = 5.00, M_W = 0.94, p = .049) on providing sample patient information sheet and consent forms with information for patients on the purpose of the PRO assessment and who has access to PRO data. The mean completeness scores for workshop protocols was higher than those of the control protocols on nine of the 18 additional PROtocol Checklist items, but the score distributions did not differ significantly.

Table 4

Mean (M) and standard deviation (SD) of the completeness rating¹ for each additional PROtocol Checklist for control and workshop protocols
PROtocol Checklist item #	Additional items	Control protocol M (SD)	Workshop protocol M (SD)	p value^2,*
2	In the protocol summary - Identify specific PRO endpoint(s), specifying key PRO construct(s)/domain(s), time-point(s), analysis metric(s). (i.e. change in score)	1.83 (1.37)	3.47 (2.81)	0.234
3	In the protocol summary - PRO assessment included in the study schema / assessment schedule.	5.00 (5.48)	5.94 (4.91)	0.706
4³	Do the stated PRO objectives/hypotheses include time-points?	0.33 (0.61)	1.28 (2.17)	0.721
13	Specify who is responsible for delivering PRO questionnaires to patients and retrieving completed questionnaires from them, or if online, who is responsible for sending reminders.	2.67 (2.66)	1.59 (2.73)	0.420
15	Specify what should be done when PRO assessments are missed, including contingency plans for following up patients who miss PRO assessments and who is responsible for implementing them.	3.75 (3.75)	2.75 (3.41)	0.313
24	Specify where PRO questionnaire data will be stored.	8.33 (4.08)	7.50 (4.47)	0.685
25	Specify security measures in place to ensure confidentiality of patient data.	6.92 (3.44)	6.75 (2.30)	0.629
26	Specify what will happen to a patient's PRO data if that patient decides to exit the study.	1.67 (4.08)	1.88 (4.03)	0.912
27³	Describe methods for deriving PRO endpoints from PRO data.	0.58 (1.20)	3.94 (3.25)	0.011*
29	Where possible, reference scoring manuals for summated scales from questionnaires (domain-specific &/or total), and methodological papers for composite endpoints (e.g. QTWiST).	0.00 (0.00)	1.88 (4.03)	0.265
30	Describe PRO responder definitions (size and duration of benefit), where relevant.	0.00 (0.00)	1.33 (3.04)	0.394
32	State minimal important difference (with reference/s) – relevant to sample size calculations, responder definitions and interpreting clinical significance of results.	0.00 (0.00)	1.88 (4.03)	0.265
35	Provide references for what is known about PROs (as per Background and Rationale section)	6.67 (5.16)	6.31 (4.92)	0.930
36	Provide references for PRO data analyses and methods for handling missing data	0.00 (0.00)	1.25 (3.42)	0.375
37	Provide copies of PRO questionnaires.	8.33 (4.08)	5.63 (5.12)	0.251
38	Provide evidence of permission to use PRO questionnaires (if permission not required, this is stated).	1.67 (4.8)	0.63 (2.50)	0.460
39	Provide copies of the Patient Reported Outcomes (PRO) Completion and Missing Data (CoMiDa) Form – to record reasons for missing PRO data, which may inform analyses	0.00 (0.00)	0.63 (2.50)	0.540
40	Provide sample Patient Information Sheet and Consent form (in which the patient is informed about the requirement and purpose of PRO questionnaires in this research, who has access to the PRO data and who to contact with questions).	5.00 (5.48)	0.94 (2.72)	0.049*
1. Higher scores indicate more completely addressed items.
2. Mann-Whitney U tests, * indicates statistical significant at 95% level (p < 0.05)
3. These represent part of PROtocol Checklist items 4 and 27; the other parts coincided with SPIRIT-PRO items 7 and 12_ii_y

We provided a 2-day PROtocol Checklist workshop designed to equip trialists with the motivation, knowledge and resources to write PRO content in trial protocols. Participant rated workshops highly and acknowledged the value of PROs and the PROtocol checklist. However, few reported using the PROtocol checklist in the long-term; barriers were staff churn and time constraints. PRO components of protocols produced after were only marginally more complete than control protocols, and were often incomplete.

The specific protocol omissions we identified potentially increase the risk of research waste in the following ways. No protocol identified an individual responsible for PROs, increasing the risk of the PROs not being analysed and reported adequately/at all. Failure to report PRO data is concerningly common, with rates ranging from 37–43% in four reviews(17–20). Multiplicity was poorly addressed; this leads to a higher chance of false positive findings, potentially misleading clinical practice and policy(21). PRO assessment time points were often not specified, with staff at recruiting sites not knowing when to collect data, increasing the risk of missing PRO data(6, 7, 9). Few protocols explained how to handle missing data, which would likely to lead to inappropriate handling of missing data and bias(9, 22). The general inadequacy of PRO content suggests PRO endpoints were prioritized by the trials groups, staff or principal investigators, so site staff training and PRO analysis may not have been appropriately budgeted for, reducing the quality of PRO data collected and the likelihood of publication. Similar deficiencies and risks have been identified in previous systematic reviews of trial protocols(13, 15).

We did not obtain all protocols brought to the workshop, so our results may not be representative. Nevertheless, our findings were disappointing after nearly a decade of delivering workshops specifically tailored to the topic and receiving glowing participant evaluations. How might future educational efforts be more effective?

First, target audience: We encouraged CCTGs to send principal investigators and trial group staff with protocol development roles to our workshops. Trial staff may not have felt they had the authority to make substantial changes to protocols, in which case the attitudes of principal investigators and trial staff managers about PRO endpoints and PRO-specific protocol content would be critical. However, recent research revealed skepticism about PROs due to their subjectivity and focus on survival outcomes among oncology trialists and principal investigators, relegating PROs to a relatively low position in the trial outcome hierarchy(23, 24). Our long-term survey revealed two challenges to building PRO expertise within trials groups: we were unable to contact about 20% of workshop participants, and 34% of responders said their current roles involve developing protocol content. Staff turnover and role change is inevitable, so ongoing workforce training is needed.

Second, workshop format and instructional methods: Long-term learning is facilitated by a series of temporally separated lessons with practice opportunities distributed within and across lessons - the ‘spacing effect’(25). Our 2-day intensive format was internally spaced; content was organised into a series of topics, with lectures punctuated by periods of reflection (small-group discussion) and practice (individual protocol writing). Worked examples are recommended to illustrate underlying principles, especially for novice learners(25). We included excerpts from oncology protocols to illustrate how to address PRO issues, but learning would have been bolstered by ‘faded’ (partially complete) examples and comparison among poor versus exemplary examples, plus practice exercises/questions for each protocol topic (25). For questions, explanatory feedback about why an answer is correct/incorrect is an important instructional method. Finally, novice learners are more subject to cognitive overload than experienced learners and may benefit from different instructional methods(25). Few of our workshop participants had previous training in PROs, and immediately post-workshop, participants could have had competing work priorities. Without repetition and reinforcement of applying the knowledge gained during the workshop, participants would likely forget the key principles and learnings.

Solving this array of issues requires a multifactorial response. Greater appreciation of the value of PROs by investigators, trial group managers and staff may motivate more investment in developing PRO expertise and prioritizing PRO content during protocol development. Educational resources are needed, tailored to the training needs of various target audiences, using evidence-based instructional methods and formats. Self-directed online modular formats provide scheduling flexibility for busy professionals. Stand-alone course could provided in Australia by organisations PRAXIS Australia(26) and Cancer Institute NSW(27), or provided by universities as part of graduate and post-graduate level in courses. Professional development incentives may be effective for short-term engagement but not necessarily for deep learning or sustained practices. An interesting new adjunct to training currently in development is an online protocol authoring tool based on SPIRIT, SEPTRE, designed to help develop protocol content, with future plans to incorporate SPIRIT-PRO guidance for PRO aspects(28). Trials groups could be encouraged to incorporate SPIRIT-PRO into their protocol templates, and trial registries and journals could require authors to comply with SPIRIT and SPIRIT-PRO. Trials groups could encourage investigators to consider PROs early in the development process, before drafting the protocol, and encourage statisticians to consider PROs using the SISAQOL guidance(21) when writing the statistical analysis plan. Again, guidance alone does not ensure compliance; motivation, training and expertise is also needed.

There are some possible explanations as to why there were few differences between the workshop and control protocols. One is that workshops were not effective, as discussed above. Another is contamination effect: we opted for contemporaneous controls rather than historical controls, meaning that the control protocols were developed during the period when workshops were being conducted, so may have been influenced indirectly by knowledge developed by CCTG staff who attended a workshop and were responsible for protocol writing for their group. Also, the PROtocol Checklist was promoted as a resource to the CCTGs and freely available via the SQOLO website. It is interesting that even though about a third of workshop respondents stated that the sessions on statistical considerations for PROs were difficult to follow for people with no background in statistics, the SPIRIT-PRO item on analysis did actually improve in completeness relative to controls, suggested that the workshop was effective on that point.

Our study had several limitations. Despite a relatively large number of participants attending the workshops (n = 107), we were able to obtain only a small sample of protocols, particularly control protocols. This was a major limitation, both in terms of the play of chance in which protocols we sampled and power to detect differences due to the workshops. However, availability was limited: of 74 protocols brought to the workshops, only 25 were finalised to study activation; QOL Office staff had contributed to nine of these, so to avoid bias only 16 protocols were included in the review. Confidentiality concerns may have limited the availability of protocols, including control protocols. There may also have been some selection bias if CCTG offered high quality protocols as controls. Another limitation was that the six control protocols were either Phase I, Phase II or correlative design, while the workshop protocols were either Phase II or Phase III; greater balance in trial phase between workshop and control protocols would have been preferable. Half of the controls were led by one investigator and trials group, which limited the variability of the control group. Another limitation was that the workshops were based on the SQOLO PROtocol Checklist but we focused our evaluation on SPIRIT-PRO; we did this because SPIRIT-PRO is more relevance to the trials community moving forward, and it was possible to do so because our checklist contained most of the 16 SPIRIT-PRO items. We did not assess workshop participant or CCTG attitudes towards the value of PRO assessment, or their training or experience with PROs, so we were not able to assess whether this was an underlying influence on how much attention PROs were given in protocols. It would also have been useful to know this about the developers of the control protocols. Finally, we did not assess supporting documents, such as site manuals, standard operating procedures (SOPs) or statistical analysis plans, all of which may have contained information on PRO endpoints to complement the trial protocol content. These supplementary documents are difficult to obtain and there is huge variation in how different recruiting sites and trials groups manage such instructions, particularly those relating to the conduct of the trial. These limitations suggest improvements for future studies with similar aim.

Writing specialist PRO content for trial protocols requires expertise, time and effort if PRO content of protocols is to meet the minimum standards set out in the SPIRIT-PRO guidance. This requires two things: appreciation by clinician researchers and trials group managers of the value of PROs endpoints in clinical trials, and investment in effective workforce training. One-off education workshops are not enough to develop needed expertise in PROs; series of easily accessible and effective educational activities, longer-term mentoring programs, and institutional requirements for use of existing resources may be needed to improve the content of trial protocol development and study design, which in turn will likely improve conduct and reporting, all reducing research waste associated with these aspects of clinical research.

Acknowledgements

We would like to thank the following people for their valuable contributions as workshop faculty members regarding statistical considerations and missing data (Melanie Bell 2011-2013, Daniel Costa 2014-2017) and health economics (Rosalie Viney 2013-2016, Richard d’abreu Laurenco 2013). We would also like to thank the following Multi-site Collaborative Cancer Clinical Trials Groups (CTGs) for providing the protocols reviewed to address the third aim of this paper: Australasian Leukaemia and Lymphoma Group (ALLG); Australasian Lung Cancer Trials Group (ALTG); Australian New Zealand Breast Cancer Trials Group (BCT); Cooperative Trials Group for Neuro-Oncology (COGNO); Psycho-Oncology Co-operative Research Group (PoCoG); Primary Care Collaborative Cancer Clinical Trials Group (PC4); Australia and New Zealand Melanoma Trials Group (ANZMTG).

Compliance with Ethical Standards

Funding

The development and provision of the PROtocol Checklist and the PROtocol Checklist workshop was funded by the Australian Government through Cancer Australia. The authors did not receive support from any organization for conducting this study or for the submitted work.

Conflicts of interest

Authors MK, MT, RC, FM, CR, and RMB received partial funding towards salaries from the Australian Government through Cancer Australia while providing the Quality of Life Technical Service to the Multi-site Collaborative Cancer Clinical Trials Groups. In their roles within the Multi-site Collaborative Cancer Clinical Trials Groups, authors CB, SC, DL, and JS received partial funding towards salaries from the Australian Government through Cancer Australia.

Authors MK and RMB were members of the executive group that led the development of the SPIRIT-PRO guidance.

Ethics approval, Consent, Data, Material

Ethical approval for this study was granted by University of Sydney Human Research Ethics Committee (HREC) Project#: 20789, and informed consent to participate in the study was obtained from all participants. This study was performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

1. Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Annals of internal medicine. 2013;158(3):200-7.

2. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet (London, England). 2014;383(9913):257-66.

3. Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet (London, England). 2014;383(9912):166-75.

4. Vodicka E, Kim K, Devine EB, Gnanasakthy A, Scoggins JF, Patrick DL. Inclusion of patient-reported outcome measures in registered clinical trials: Evidence from ClinicalTrials.gov (2007-2013). Contemporary clinical trials. 2015;43:1-9.

5. Mercieca-Bebber R, Williams D, Tait MA, et al. Trials with patient-reported outcomes registered on the Australian New Zealand Clinical Trials Registry (ANZCTR). Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation. 2018;27(10):2581-91.

6. Mercieca-Bebber R, Calvert M, Kyte D, Stockler M, King MT. The administration of patient-reported outcome questionnaires in cancer trials: Interviews with trial coordinators regarding their roles, experiences, challenges and training. Contemporary clinical trials communications. 2018;9:23-32.

7. Kyte D, Ives J, Draper H, Keeley T, Calvert M. Inconsistencies in quality of life data collection in clinical trials: a potential source of bias? Interviews with research nurses and trialists. PLoS One. 2013;8(10):e76625.

8. Fielding S, Ogbuagu A, Sivasubramaniam S, MacLennan G, Ramsay CR. Reporting and dealing with missing quality of life data in RCTs: has the picture changed in the last decade? Quality of Life Research. 2016;25(12):2977-83.

9. Palmer MJ, Mercieca-Bebber R, King M, Calvert M, Richardson H, Brundage M. A systematic review and development of a classification framework for factors associated with missing patient-reported outcome data. Clinical Trials. 2018;15(1):95-106.

10. Fielding S, Maclennan G, Cook J, Ramsay C. A review of RCTs in four medical journals to assess the use of imputation to overcome missing data in quality of life outcomes. Trials. 2008;9(1):51.

11. Mercieca-Bebber R, Friedlander M, Calvert M, et al. A systematic evaluation of compliance and reporting of patient-reported outcome endpoints in ovarian cancer randomised controlled trials: implications for generalisability and clinical practice. Journal of patient-reported outcomes. 2017;1(1):5.

12. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet (London, England). 2009;374(9683):86-9.

13. Kyte D, Duffy H, Fletcher B, et al. Systematic Evaluation of the Patient-Reported Outcome (PRO) Content of Clinical Trial Protocols. PLoS ONE. 2014;9(10).

14. Kyte D, Retzer A, Ahmed K, et al. Systematic Evaluation of Patient-Reported Outcome Protocol Content and Reporting in Cancer Trials. Journal of the National Cancer Institute. 2019;111(11):1170-8.

15. Mercieca-Bebber R, Friedlander M, Kok PS, et al. The patient-reported outcome content of international ovarian cancer randomised controlled trial protocols. Quality of life research : an international journal of quality of life aspects of treatment, care and rehabilitation. 2016;25(10):2457-65.

16. Calvert M, King M, Mercieca-Bebber R, et al. SPIRIT-PRO Extension explanation and elaboration: guidelines for inclusion of patient-reported outcomes in protocols of clinical trials. BMJ Open. 2021;11(6):e045105.

17. Lombardi P, Marandino L, De Luca E, et al. Quality of life assessment and reporting in colorectal cancer: A systematic review of phase III trials published between 2012 and 2018. Critical reviews in oncology/hematology. 2020;146:102877.

18. Marandino L, La Salvia A, Sonetto C, et al. Deficiencies in health-related quality-of-life assessment and reporting: a systematic review of oncology randomized phase III trials published between 2012 and 2016. Annals of oncology : official journal of the European Society for Medical Oncology. 2018;29(12):2288-95.

19. Reale ML, De Luca E, Lombardi P, et al. Quality of life analysis in lung cancer: A systematic review of phase III trials published between 2012 and 2018. Lung cancer (Amsterdam, Netherlands). 2020;139:47-54.

20. Schandelmaier S, Conen K, von Elm E, et al. Planning and reporting of quality-of-life outcomes in cancer trials. Annals of oncology : official journal of the European Society for Medical Oncology. 2015;26(9):1966-73.

21. Coens C, Pe M, Dueck AC, et al. International standards for the analysis of quality-of-life and patient-reported outcome endpoints in cancer randomised controlled trials: recommendations of the SISAQOL Consortium. The Lancet Oncology. 2020;21(2):e83-e96.

22. Mercieca-Bebber R, Palmer MJ, Brundage M, Calvert M, Stockler MR, King MT. Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review. BMJ Open. 2016;6(6).

23. Retzer A, Calvert M, Ahmed K, et al. International perspectives on suboptimal patient-reported outcome trial design and reporting in cancer clinical trials: A qualitative study. Cancer Medicine. 2021;00:1-13.

24. Mercieca-Bebber R, King MT, Calvert MJ, Stockler MR, Friedlander M. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient related outcome measures. 2018;9:353-67.

25. Clark RC. Evidence-based training methods: a guide for training professionals. Appendix: A Synopsis of Instructional Methods. Third edition. ed. Alexandria, VA: ATD Press; 2020.

26. PRAXIS Australia. PRAXIS Australia: Promoting Ethics and Education in Research 2021 [Available from: https://praxisaustralia.com.au/.

27. Education e. The eviQ Education program and eviQ Education website 2020 [Available from: https://education.eviq.org.au/.

28. Chan A-W, Et al. SPIRIT Electronic Protocol Tool and Resource (SEPTRE) 2020 [Available from: https://www.spirit-statement.org/trial-protocol-template/.

Box 1. PROtocol Checklist workshops: Topics covered each year, Format, and Resources.

Workshop Session Topics	Workshop Year
	2011	2012	2013	2014	2015	2016	2017
Introduction, Rationale, Objectives and Hypotheses	x	x	x	x	x	x	x
PRO measures	x	x	x	x	x	x	x
Utility measures for health economic evaluation			x	x	x	x	x
PRO questionnaire administration	x	x	x	x	x	x	x
Missing data - statistics and logistics	x	x	x	x	x	x	x
Endpoints and Statistical Considerations	x	x	x	x	x	x	x
Interpretation and Clinical significance of PRO results				x	x	x	x
Data quality assurance, appendices, and resources	x	x	x	x
Workshop format: The PROtocol checklist guided the structure of each workshop. For each topic, the general principles were explained in a lecture that included examples, either from real protocols or research publications. Each lecture was followed by a small-group discussion, facilitated by faculty, focussed on how the general principles were applied to the participants’ protocols. Workshop material was refined each year based on contemporary research and workshop participant feedback.
Workshop resources: Workshop participants were given a booklet that included the lecture slides, a bibliography of all papers referenced in the slides, excerpts from oncology protocols to illustrate how to address specific PROtocol Checklist items, and the PROtocol Checklist. See online supplement 1 for booklet table of contents and workshop program.

Box 2. Examples of post-workshop feedback comments.

Typical post-workshop feedback comments

“Good comprehensive overview of QOL measures, incorporation into a protocol and other factors to consider. Good practical examples and good to be introduced to the protocol checklist”. – ID72

“It was very helpful and made me feel more comfortable about including PROs in my studies”. – ID16

“I will use the checklist for each new protocol and share it with principle investigators”. – ID17

“The checklist is very useful – makes you think of things that might otherwise slip through the cracks”. – ID24

Comments providing constructive criticism

“Stats session was a bit in-depth for what we would use in our roles” -ID75

“more interaction - as a lot of didactic content” – ID37

Improving the patient-reported outcome sections of clinical trial protocols: a mixed methods evaluation of educational workshops

Status:

Version 1

Abstract

Figures

Introduction

Methods

The PROtocol Checklist Workshop

Aim 1 – Post-workshop evaluation survey (immediate)

Aim 2 – Long-term research practice survey

Aim 3 – Completeness of PRO content of protocols

Analysis

Results

Participants and post-workshop evaluation survey

Follow-up survey

Protocol Evaluation

Discussion

Conclusions/implications

Declarations

References

Box

Supplementary Files

Status:

Version 1