1.3.1 Comprehensive literature review of previous translation approaches
346 records were identified from the COMET database from which four studies (summarized in table 1) were deemed eligible for inclusion in the comprehensive literature review. The process through which these were identified are summarised in figure 1.
1.3.1.1 General Approaches to Translation currently in use
All 4 COS groups summarized their approach to translating surveys with one providing a reference to their methodology and another referring to methodology described by the OMERACT group at a COS development meeting 9,12. In addition to forward translations, three groups undertook a backward translation of the survey from the target to source language. The number of forward and backward translations differed in each study. Two studies undertook a single forward translation whilst the others undertook two and three. One group used no backward translations, one study undertook a single backward translation whilst the other two undertook two backward translations. The characteristics of those involved in the translation processes also differed amongst the groups (figure 2); no paid translation services were employed, and all translations were undertaken by healthcare professionals or lay translators.
Table 5.1 Studies using multi-language Delphi surveys in the development of international COS.
Condition/Group
|
Original Language
|
Target Language(s)
|
Total Participants in Surveys
|
Total Participants Using Translated Survey(s) (%)
|
Hip and Knee Osteoarthritis
OMERACT-OARSI13
|
English
|
Italian & Spanish
|
426
|
2 (0.5%)
|
Medication review in multi-morbid older patients with polypharmacy
OPERAM14
|
French
|
Dutch, German, English
|
150
|
118 (79%)
|
Idiopathic inflammatory myopathy
OMERACT15
|
English
|
Swedish, Dutch & Korean
|
500
|
120 (24%)
|
Vascular Malformations
OVAMA Group16
|
English
|
Dutch
|
301
|
72 (24%)
|
Gastric cancer
GASTROS Study
|
English
|
Chinese, Dutch, German, Italian, Portuguese, Spanish, Turkish,
|
952
|
637 (66%)
|
1.3.1.2 Discrepancies and Harmonization
All four groups described an approach to managing discrepancies in translations. Two groups reported that discrepancies were discussed within the ‘research group’ until consensus was reached, whilst the remaining two referred to individuals outside of the ‘research group’ who were fluent in the target language to resolve any language issues.
1.3.1.3 Cognitive Debriefing
Three groups described undertaking an exercise to test alternative wording and check understandability, interpretation, and cultural relevance of the proposed Delphi survey in the target language. Using interviews, they studied patients/relatives and health professionals’ interpretation of the translations to examine face validity (the degree to which the survey appears effective in terms of its stated aims). Two of these involved patients and/or their relatives whilst the third was based on the opinion of healthcare professionals alone.
1.3.2 Results from GASTROS Study
The GASTROS study was able to recruit 952 eligible participants (445 surgeons, 268 patients, 239 nurses) in the first round of the Delphi survey, with 315 participants using the English language version and 637 using one of the seven other language versions (table 2). 62% (166/268) of patients used translated surveys compared to 69% (471/684) of healthcare professionals (62% of surgeons and 82% of nurses).
1.3.2.1 Development of translation approach
Below, we describe ten steps involved in translating the Delphi survey used in the development of a COS for surgical trials in gastric cancer. The full rationale for each step, and the risks of omitting them, is described in detail in the ISPOR-TCA guidance; we have stated the rationale for the steps below (particularly in relation to pragmatic deviations) where we believed it was necessary to do so.
Supplementary appendix 2 details the instructions which were provided to each international collaborator responsible for leading the translation process in their respective country. These outlined which files required translation, how the translation should be undertaken, and by whom.
1.3.2.1.1 Step 1: Preparation
a. Cognitive Debriefing:
i. Cognitive debriefing describes a process which aims to identify issues with comprehensibility of key concepts and understanding amongst potential participants. As previously stated, we presented survey participants with 56 outcomes which had been rationalized following a process that had identified a long-list of potentially important outcomes from a systematic review and in-depth patient interviews. The rationalization process from the long-list to the 56 survey items involved key-stakeholders (members of the GASTROS study group, surgeons, oncology nurses and patients) who also ensured that the outcomes were accompanied with plain English-language explanations that could be understood by all participants including patients. A further consultative exercise with an English-speaking patient-group was held to ensure that the meaning of each outcome, in addition to other survey-related files were clearly understood. Undertaking this work prior to translation was essential as it minimized the possibility of ambiguous meanings which could result in a mistranslation.
Table 5.2 Uptake of translated Delphi surveys in non-English speaking regions.
Regional Language
|
Patients (n=268)
|
Surgeons (n=445)
|
Nurses (n=239)
|
|
Translated Version
|
English Version (%)
|
Translated Version (%)
|
English Version (%)
|
Translated Version (%)
|
English Version (%)
|
Chinese
|
60 (97%)
|
2 (3%)
|
109 (97%)
|
3 (3%)
|
109 (100%)
|
0 (0%)
|
Dutch
|
5 (100%)
|
0 (0%)
|
22 (100%)
|
0 (0%)
|
10 (100%)
|
0 (0%)
|
German
|
4 (100%)
|
0 (0%)
|
10 (100%)
|
0 (0%)
|
-
|
-
|
Italian
|
57 (100%)
|
0 (0%)
|
57 (95%)
|
3 (5%)
|
12 (100%)
|
0 (0%)
|
Portuguese
|
1 (100%)
|
0 (0%)
|
28 (88%)
|
4 (12%)
|
8 (100%)
|
0 (0%)
|
Spanish
|
-
|
-
|
33 (94%)
|
2 (6%)
|
0 (100%)
|
0 (0%)
|
Turkish
|
39 (100%)
|
0 (0%)
|
17 (94%)
|
1 (6%)
|
56 (100%)
|
0 (0%)
|
Other language*
|
No translation undertaken.
|
0
|
No translation undertaken.
|
97
|
No translation undertaken.
|
13
|
TOTAL
|
166 (99%)
|
2 (1%)
|
276 (96%)
|
13 (4%)
|
195 (100%)
|
0 (0%)
|
Percentages reported refer to the proportion of participants from the respective region within each stakeholder group. *‘Other language’ refers to regions where English was not the first language, but where the survey was not translated.
b. Preparing documents for translation
i. Four documents were needed to run the Delphi survey; a participant information sheet and three further files which were required to set up the web-based survey. We used DelphiManager 3.0 platform, developed and maintained by the COMET Initiative (http://www.comet-initiative.org/), to undertake the Delphi survey (see ‘important considerations’). The three files included:
- File 1 (appendix 3): An excel file containing details of each outcome, accompanying meaning and the ‘outcome area’ under which the outcome was categorised10.
- File 2: User-defined text: A file containing text specific to our surveys (in this case the GASTROS Delphi survey).
- File 3: Static text: A file containing text common to all Delphi surveys which was used in the setting up process by the DelphiManager team.
ii. Preparation for Round 2 of the Delphi survey: Additional translations were required to support the second round of the survey. These included:
- Outcomes identified by participants in round 1 as being important to consider that were not identified from the systematic review or patient interviews.
- Legends and terms required to produce charts which were presented to survey participants in round 2.
- Comments and feedback from study participants.
ii. Following Round 2 of the survey:
- Participants who changed their scores between rounds were given the opportunity to provide their reasons for doing so.
- Participants were also given the opportunity to provide further comments after completing the survey.
c. Understanding which methodological approaches to employ
Due to the resources required for different methodologies, we opted for two approaches to translation. Our rationale for applying each approach is described below:
- “Two forward, one back translation”; the terms ‘forward’ and ‘backward’ refer to the direction of translation between the source and target languages, with forward referring to a translation from the source language and backward referring to a translation from the target language back to the source language. This approach was the most comprehensive and labour intensive as it required a further nine steps (below) before a final file version was agreed. Following discussion amongst the study management team, it was deemed content which could alter the meaning of the outcomes being presented and ultimately influence how the overall aims of the survey was received and understood by participants (file 1, file 2, and additional outcomes identified by participants in round 1) underwent this approach. The steps involved in this process are described in greater detail in points 2 to 10.
- “One forward, dual independent proofreading”; File 3 consisted primarily of short instructional phrases (e.g. ‘click here’, ‘register’, and ‘next page’) which were necessary for the functionality of the survey. As these terms would not materially influence the comprehension of the survey’s purpose or outcomes presented within it, a simplified, less resource intensive approach was adopted. This file underwent a single forward translation followed by two independent proof-readings by translators who compared the translated and source files for accuracy and quality. Any corrections or amendments were undertaken through discussion between the translator and proof-readers. This approach was also adopted for the translation of participant comments, feedback and reasons for changing scores between round 1 and round 2.
d. Setting up translation teams
To support the translation work, an international working group (IWG) was established (see ‘important considerations’). Each collaborator within the IWG was responsible for overseeing a team which would undertake the translation and ensuring that the key concepts of the study were appropriately communicated. The translation process was supported by the GASTROS study Chief Investigator (BA) if any clarifications were required. The characteristics of individuals involved in this process are described in greater detail in supplementary appendix 2. In summary, each team was made up of an IWG lead, two forward translators and a single backward translator.
e. Developing instructions for translations
Setting out the methodology a priori in a clear and structured document ensured that collaborators and their teams understood what would be required of them at each stage of the translation. These instructions included ongoing responsibilities prior to and following future rounds of the Delphi survey. This was essential given that one of our primary aims was to ensure that our approach was easily replicable. Figure 3 is a flow diagram which details these stages and the order in which they were to be undertaken. Feedback from the IWG was positive in response to these instructions with collaborators reporting that the document enabled them to undertake the translation process efficiently.
f. Quality assurance
IWG collaborators were asked to provide documented evidence for each step of the translation process. These could then be reviewed by the study management team as required.
1.3.2.1.2 Step 2: Forward Translation
Two independent forward translations by individuals who were native speakers of the target language were undertaken. Culture is a primary determinant of language and therefore native speakers have advantages with language abilities compared to second-language speakers. Having two independent forward translations enables detection of errors and divergent interpretations that could otherwise lead to bias.
1.3.2.1.3 Step 3: Reconciliation
There are several approaches which can be used to reconcile the forward translations. We opted to use the ‘in-country’ IWG collaborator who was also involved in cognitive debriefing and piloting of the survey as this was pragmatic and would not require the identification of further individuals to undertake this step. No issues arose from the reconciliation process, however, had further clarifications been required, they would have been directed to the Chief Investigator (CI).
1.3.2.1.4 Step 4: Back Translation
The issue of whether ‘back translation’ is required is one on which there is disagreement; the ISPOR-TCA guidance states that ‘back translation’ is necessary, whilst the Survey Research Centre guidance suggests that it is not. COS developers may therefore be justified in omitting steps 4 and 5 of our approach. This should however be done after careful consideration as the importance of back translation may depend on the type of outcomes that are being translated. It is possible that certain outcomes are conceptually alien between cultures or geographical regions and undergoing an added step to reduce the risk of mistranslation is warranted. In the field of patient-reported outcome measurement (PROM), it is common for questionnaires to undergo translations (for use in international trials). The methods required for PROM translation is rigorous and includes back translation11. Whilst it may be argued that less rigorous methods could be used in Delphi surveys for COS, to ensure optimal face validity of items the same standards are recommended.
We opted to undertake a single back translation to provide quality-control of the forward translations. Whilst the ISPOR-TCA guidance suggests that this should be undertaken by individuals who are native speakers of the source language (i.e. English), we found it challenging to identify seven native English-speakers who were also fluent in the required target languages and had an understanding of outcome reporting without referring to a professional service (paid professionals with expertise in translation). We opted to ensure that back-translators were fluent in English and independent from the forward translators.
1.3.2.1.5 Step 5: Back Translation Review
This step is important as it ensures that the cross-cultural adaptation needs of the translation is met. Cross-cultural adaptation ensures that the imprinted knowledge, attitudes, values, perceptions and behaviours of different regions are accounted for in the understanding of the terms being translated. Without it, there is a risk of that a mistranslation or omission would remain in the translation. This was undertaken by the CI in combination with the IWG collaborator by comparing the back translation to the source document. No significant discrepancies between the source and back-translated files were identified across any of the translations.
1.3.2.1.6 Step 6: Harmonization Across Different Languages
There is no agreed method to how harmonization across different translations should be enforced; many approaches omit this step. However, our group opted to ensure harmonization between each language at each step of the process. This was undertaken by the CI. We did not encounter significant differences between translations. An example of a minor change that was made across surveys was the term ‘last round scores’ which in the context of the survey meant ‘previous round scores’. Some teams translated this as ‘the final round scores’ which had to be altered to ensure all versions contained the same meaning.
1.3.2.1.7 Step 7: Cognitive Debriefing of the Translation
Following harmonization across translations, all survey versions were built using the DelphiManager platform (see ‘important considerations’). A further cognitive debriefing exercise was undertaken by asking IWG collaborators and their translation teams to complete a pilot version of the survey to identify grammatical or stylistic errors and check understandability, interpretation and cultural relevance of instructions and outcomes within the survey.
1.3.2.1.8 Step 8: Review of Cognitive Debriefing Results and Finalization
There were no issues highlighted with comprehensibility or understanding. Spelling mistakes and minor grammatical errors (e.g. pronouns ‘you’ formal and informal) were altered.
1.3.2.1.9 Step 9: Proofreading
IWG collaborators were once again asked to examine the survey and ensure that any issues highlighted in the previous steps had been addressed. No further changes were identified in any of the language versions by this stage.
1.3.2.1.10 Step 10: Final Report and ‘Start of Survey’
The ISPOR-TCA group guidance recommends that a report should be produced detailing the methodological approach for translation and rationale for each step. The final report for translations undertaken for the GASTROS study is represented by this paper.
1.3.3 Important Considerations
Whilst applying the described approach to translating the GASTROS Delphi survey, several key issues were identified that are summarised in table 3 and described in greater detail below. These should be considered alongside the translation work to maximise recruitment. We describe the rationale for each consideration and the potential risks of not applying these steps (where applicable).
1.3.3.1 International Working Group
The GASTROS study is a collaborative international initiative which sought to attract global representation within the study group. Motivated, research-active collaborators from countries with a significant incidence of gastric cancer were approached to form an IWG. Individuals signed a ‘terms of reference’ document which outlined the benefits of their involvement in addition to the following responsibilities:
- To form a local team and oversee the translation of the Delphi survey (where applicable)
- To drive recruitment locally, regionally, nationally, and internationally through organisations and personal networks
- To garner and develop links specifically with patient groups who would be able to participate in advertising the Delphi survey
- To identify the need and apply for relevant local ethical and regulatory approvals
Table 5.3 Nine key considerations for COS developers undertaking multi-language Delphi surveys.
- International working group
|
To ensure that study and its aims are promoted in regions from where the study team wish to target recruitment.
|
- Patient and public involvement
|
To ensure that the patient perspective is represented.
|
- Who should undertake the translation work?
|
Deciding whether to employ professionally paid services or identify clinically trained individuals to undertake the translations.
|
- Milestones and timelines planning
|
Providing a pre-agreed timetable for translation work and checks ahead of recruitment to the Delphi survey.
|
- Recruitment and retention targets
|
Planning how long to keep Delphi survey rounds open to ensure an appropriate number of participants have been recruited.
|
- Paper and internet-based survey versions
|
Giving stakeholders without easy access to the internet an opportunity to participate in the study.
|
- Measures to maximise recruitment
|
Dissemination strategy
Local recruitment
Support from stakeholder group and research networks
Collaborations
Personalised emails
Social media and multimedia
|
|
Identifying what type of approvals are required as these vary between regions.
|
|
Ensuring that a robust plan for resource allocation is made in advance.
|
The IWG was made up of collaborators from the following countries:
- Brazil
- Mainland China and Hong Kong
- Germany
- Ireland
- Italy
- Japan
- The Netherlands
- Nigeria
- Portugal
- South Korea
- Spain
- Turkey
- United Kingdom
Ensuring the IWG was set up early maximized our ability to develop translations in a timely manner and recruit evenly across all stakeholder groups from a broad range of countries.
1.3.3.2 Patient and Public involvement
A Study Advisory Group (SAG) separate to the IWG formed part of the management structure of the wider GASTROS study. The SAG was made up of key stakeholder representatives including patients. The group provided advice on the methodology of the study, general delivery of the study against its stated objectives and ensured that the viewpoints of all stakeholder groups were considered. In addition, patient groups (see acknowledgements) were vitally important in reviewing and piloting the translated surveys prior to recruitment to the Delphi. These groups were also instrumental in recruiting patients (see below).
1.3.3.3 Who should undertake the translation work?
The GASTROS study management group opted to set up local translation teams made up of healthcare professionals who met the rigorous criteria as set out by the ISPOR-TCA group. An alternative approach would have been to employ a professional translation service to undertake this work. One of the benefits of professional services is the ability to complete the translations in a relatively short period of time, in addition to developing an unlimited number of translations which may have resulted in wider participation in the Delphi survey. The main disadvantage to this approach is cost. Quotes from three different professional translation services (all familiar with the ISPOR-TCA guidance) were requested to support rounds 1 and 2 of the survey. In April 2018, the estimated costs were in the region of 3200GBP-4000GBP per language. All translations for rounds 1 and 2 of the survey would be finalized within 5 and 2 weeks respectively. Due to the financial limitations of undertaking the survey in 7 languages, we did not pursue this option.
1.3.3.4 Milestone & timeline planning
A summary of the resulting timelines involved in producing all versions of the survey using our approach to translation is provided in table 4. Setting aside enough time for the translation process is of paramount importance, particularly if COS developers are seeking to translate their surveys into more than one language. Some of the translation steps required all language versions to have reached the same stage prior to moving onto the next stage. For example, all initial translations had to have been completed before harmonization across surveys could be achieved. Without this, we were unable to ask collaborators and their teams to pilot their respective surveys. Furthermore, we chose to open recruitment to all language versions simultaneously and so all translations needed to have been fully completed before participants could complete their surveys. This was also the case for the second survey round. The impact of ethical approval applications on timelines is discussed below in greater detail. The time to return the initial translation documents and obtain ethical approvals resulted in the greatest variations with respect to the overall timelines. We found that setting regular milestones and realistic timelines helped achieve the required translation objectives. Regular communication between the CI and collaborators underpinned this process.
Our aspiration was to translate the Delphi survey into Japanese and Korean to enable wider patient participation from these countries. Due to challenges in identifying collaborators at an early stage, assembling a translation team and meeting timelines, this could not be pursued. However, potential participants were invited to complete the English language version of the survey.
1.3.3.5 Recruitment and retention targets
COS developers should consider minimum recruitment targets. Whilst there is no sample size requirement for Delphi surveys, the GASTROS protocol initially set a conservative target of 100 participants in total to be recruited over a period of 6 to 8 weeks in round 1. However, as interest in the study and international collaboration grew, it was clear to see that this target would easily be surpassed. As described below, once the survey opened and momentum began to gather, we witnessed a ‘snowballing’ effect amongst all three stakeholder groups. We therefore extended recruitment to 13 weeks by which time new participation had plateaued (figure 4).
Table 5.4 Timeline related considerations in undertaking multi-language Delphi survey in the GASTROS study.
Language Version*
|
Document Preparation
|
Time to return completed translations for r1
|
Harmonization across language versions
|
Time to set up online surveys
|
Time to pilot survey and complete amendments
|
Time to obtain ethical approval**
|
Time r1 Open
|
Time to analyse results from r1 and produce additional translation files
|
Time to return translation documents for r2
|
Time r2 Open
|
Translation 1
|
2 weeks
|
6 weeks
|
2 weeks
|
1 week
|
1 week
|
25 weeks
|
13 weeks
|
3 weeks
|
2 weeks
|
12 weeks
|
Translation 2
|
10 weeks
|
1 week
|
29 weeks
|
3 weeks
|
1 week
|
Translation 3
|
3 weeks
|
1 week
|
Not required
|
3 weeks
|
1 week
|
Translation 4
|
10 weeks
|
1 week
|
Not required
|
3 weeks
|
1 week
3/7/19
12/7/19
|
Translation
|
18 weeks
|
1 week
|
***Not received
|
3 weeks
|
1 week
|
Translation 6
|
12 weeks
|
3 weeks
|
40 Weeks
|
3 weeks
|
1 week
|
Translation 7
|
2 weeks
|
1 week
|
2 weeks
|
3 weeks
|
1 week
|
* The language versions are anonymized. **This represented the time the study management group requested collaborators to begin ethical approval applications until IRB approval was received and not necessarily the time between actual submission of the application and receiving approvals. ***Ethical approval was not received before the end of round 1 of the Delphi survey. No patients were recruited from this team’s country.
For round 2, an initial retention target of 80% was set following discussions with members of our study management group who have extensive knowledge and experience of COS development. Automated reminder emails were sent out on a weekly basis to participants and support from professional bodies (in countries where round 2 responses were slow) was sought to encourage completion of the survey. Personalised e-mails from the CI to professionals were also sent. Using this strategy, we were able to retain 70% of participants from round 1 by week 13, by which time no further responses were being received.
1.3.3.6 Paper and Internet-based Delphi survey versions
The GASTROS study used both internet-based and paper versions of the Delphi survey. The internet-based versions enabled us to reach participants in nearly 60 countries, the vast majority of which did not have formal IWG collaborators. The paper-versions (printed versions of the internet-based survey which were uploaded electronically by local collaboration teams) also enabled us to recruit participants (particularly patients) who either did not readily have access to the internet or were not ‘electronically-literate’.
Several platforms exist to enable COS developers to run Delphi-surveys. These include platforms specifically designed for Delphi surveys and other generic survey platforms which researchers can use. When considering multi-language surveys, it is essential to ensure that the servers on which the surveys are hosted meet the necessary data protection regulations and are accessible particularly from countries where restrictions to certain domains exist. Furthermore, COS developers must ensure that the platforms used are able to run surveys using different language scripts and writing systems.
Our group used DelphiManager as it fulfils the required data protection criteria (as set out by our United Kingdom ethical approval) and can work with all language systems including English, Chinese, Japanese and Korean. Furthermore, the online survey domains are accessible from countries which commonly restrict access to other foreign domains. DelphiManager has additional features which simplified recruitment and completion of the surveys such as being able to send automated reminders to individuals who had yet to complete all their answers.
1.3.3.7 Measures to maximize survey recruitment
One of the strengths of our Delphi survey was that it was able to recruit approximately 1000 eligible patients and healthcare professionals from nearly 60 countries in round one. From the study’s inception, the study team recognised the importance of developing a clear networking and dissemination strategy5. We hypothesised that this was necessary to achieve broad stakeholder participation both nationally and internationally. Several strategies were employed to maximise recruitment:
1.3.3.7.1 Dissemination of results from previous study stages
The study protocol and findings from previous study stages3,5,6 were presented at targeted national and international meetings which were well-attended by potential healthcare worker participants. This was integral to generating interest and support for our study and ensured that participants understood the premise for GASTROS long before the Delphi survey opened for recruitment. All presentations contained directions to the study website and social media accounts (see below).
1.3.3.7.2 Local recruitment of patients through outpatient clinics
Ethical approval enabled the study team to recruit patients directly from outpatient clinics. Our experience from the United Kingdom is that many patients regularly attend patient support groups and are in contact with other eligible patients. As a result, a snowballing effect resulted in patients being recruited by patients already within the study.
1.3.3.7.3 Support from stakeholder groups/associations and national research networks
Support from national and international professional associations and organisations was sought in the early stages of the study. Study group members presented the study objectives at closed executive level meetings to gain support and adoption from influential bodies including professional associations, patient groups and charities. Many of these organisations have large memberships (and corresponding electronic mailing lists) through which the study was advertised. Most of the groups through whom we sent out invitations followed up an initial e-mail with a further reminder approximately 4 weeks later resulting in further recruitment spikes. Furthermore, the GASTROS study was adopted onto the National Institute for Health Research (NIHR) Portfolio (CPMS study ID 38318). This enabled us to advertise the study to healthcare professionals and patient support groups within the United Kingdom through the national Clinical Research Networks. Our experience suggests that recognition by respected associations and groups results in a ‘snow-balling’ effect with subsequent support from others becoming easier to harness.
1.3.3.7.4 Collaborations
Standardising the reporting of outcomes can be achieved through several approaches. The GASTROS study aims to identify important outcomes across the entire spectrum of outcome types. Others have concentrated on the reporting of outcomes within a defined area. For example, the Gastrectomy Complications Consensus Group (GCCG - www.gastrodata.org) have sought to standardise the reporting of all major post-gastrectomy complications18. Whilst the goals of both studies are different, both teams have been able to work closely to minimise duplication of work. In addition, the GCCG was able to promote recruitment to the GASTROS Delphi survey through its membership and respective networks. Such collaborations will also be vital for the future development of outcomes research within the field of gastric cancer surgery.
1.3.3.7.5 Personalized emails
a. Most of the study management group, study advisory group and international working group members have extensive research experience within the field of gastric cancer surgery. Each member was asked to promote the study through their personal research and clinical networks. Bulk e-mails through professional bodies may be ignored by potential participants or diverted into ‘spam’ e-mail folders, hence why this approach was employed.
b. Corresponding email addresses for authors from previous trials and protocols included in our systematic review3 were identified and personal invitations sent. This captured research-active healthcare professionals from non-English speaking regions where no formal national gastric cancer associations exist (e.g. Eastern Europe).
1.3.3.7.6. Study website, social media and multimedia
a. The study website (http://www.gastrosstudy.org) provides detailed information about the GASTROS study aims as well as all its outputs. Prior to the commencement of the Delphi survey, potential participants who had heard about the study were able to register their interest to participate in the Delphi survey. In the preceding 18 months before the survey opened, 150 healthcare professionals and patients had registered.
b. In addition to the study’s twitter account (https://twitter.com/GASTROSStudy), members of the research team posted updates on their personal Twitter and LinkedIn accounts. Regular study updates provided potential participants with an opportunity to better understand the study aims and keep up to date with its progress. Examination of analytics revealed that Twitter and LinkedIn posts in the run-up to and during round 1 of the survey regularly received over 4000 and 3000 views, respectively.
c. A series of short videos were produced for the study. These provided potential participants with an alternative way to engage with the study. At the time of writing, these videos had been viewed over 600 times. In addition to an introductory video on the study, a detailed step-by-step guide to completing the online Delphi survey was developed. This created additional content for social media platforms and the GASTROS website which in turn enabled the study to maintain a regular online presence. COS developers may wish to produce different language versions or translate video captions relatively easily to expand their reach. Additional COS-related material is already available from the COMET initiative YouTube site19 with versions available in Dutch, Portuguese and Chinese. Work is underway to develop other language versions as well.
Whilst advertising the study through these avenues increased the number of recruits, care must also be taken that potential participants are not ‘bombarded’ with requests to participate in the survey. A small number of healthcare professionals highlighted that this was an issue. This coupled with the well-recognised challenges of ‘survey-fatigue’, may in fact be counter-productive and result in apathy amongst potential participants.
1.3.3.8 Ethical Approval
The requirement for regulatory or ethical approval varied across different regions. In the United Kingdom, the approach to ethical approval has not been consistent; our group was asked to submit a full application for ethical approval committee consideration, whilst other groups have been able to gain approval through proportionate review20. Each IWG collaborator was responsible for understanding local requirements and applying for approvals if they were required. They were asked to enquire about these at the start of their agreement to participate in the study and applications were made in parallel to the translation work. Two of our international collaborating centres did not require ethical approval as local collaborators did not recruit patients directly from their clinical practice but instead advertised the study through local patient groups and recruited healthcare professionals by advertising through national Societies and networks. The time taken to complete this process varied significantly (table 4) and was largely dependent on the frequency of and access to ethics committee meetings, requirements to amend submitted materials and delays in final decisions reaching the collaborators. COS developers should investigate the need for ethical approval as early as possible to avoid unnecessary delays.
1.3.3.9 Financial Planning
Several aspects of undertaking multi-language Delphi surveys may incur significant costs depending on which approaches are adopted. COS developers should take these into account when planning their studies. These include:
- Cost of professional translations; this represents the largest financial burden and has been discussed above.
- Ethical and regulatory approvals; some of our non-UK ethical approval applications required payments of up to 250 Euros.
- Use of electronic mailing lists; some stakeholder groups may charge administration fees to send out invitations to their membership.
- Cost of Delphi survey platform; whilst open-access platforms exist, our group opted to pay to use a dedicated Delphi survey platform designed for the development of COS.
- Statistical and qualitative methods support may be required when analysing scores in rounds one and two, depending on the nature of feedback to be given.