Seven participating organisations were involved, covering six topic area collaborations in living guidelines. Three collaborations focused on COVID-19, with two on treatments and one on post COVID-19 syndrome/condition. One collaboration was in type 2 diabetes, one in rheumatoid arthritis, and one in stroke (Table 1). One organisation is a consortium (ALEC) that included four collaborators: the National Clinical Evidence Taskforce (NCET) (treatments for COVID-19 and post-acute COVID-19), Australian Living Guidelines for Diabetes, Australia and New Zealand Musculoskeletal (ANZMUSC) Clinical Trials Network, and the Stroke Foundation.
Table 1
Living guidelines collaborations
| Condition | Collaborators | Timing | Type | Duration | Description of the collaboration |
| Rheumatoid arthritis | Canadian Rheumatology Association | Collaboration planned from guideline inception | Coalition | 1–2 years | The base reviews are done through Cochrane, where the organisations collaborated directly on the reviews and living network meta-analysis. For the guideline itself, a GRADE adolopment approach was used, where the workload was shared in terms of the evidence reviews and the GRADE evidence profiles and EtDs are shared, but then the organisations developed and used the adolopment approach for their own guidelines. Rheumatology is a small community, so the organisations were aware of all the other major guideline groups and have explored other collaborations. |
| Australia and New Zealand Musculoskeletal (ANZMUSC) Clinical Trials Network (Australia Living Evidence Consortium) |
| Post COVID-19 syndrome/ condition (Long COVID) | World Health Organization | Organisations joined collaboration during guideline development | Coalition | Over 2 years | The collaboration was identified through pre-existing connections and evolved but also remained flexible. The purpose was to share evidence and more generally to share approaches and learning on the emerging health condition and approaches for assessment, diagnosis and management. Meetings are held every two months via virtual platform. At the meetings members provide updates on emerging topics and issues, changing methods or new outputs related to post-COVID condition. In between meetings members share searches, evidence summaries and reports via email. Other organizations internationally are also involved in this collaboration but were not included in this paper as they were not actively producing living guidance when this paper was being developed. |
| National Institute for Health and Care Excellence (NICE) |
| Danish Health Authority |
| National Clinical Evidence Taskforce (Australia Living Evidence Consortium) |
| Type 2 diabetes | National Institute for Health and Care Excellence (NICE) | Collaboration planned from guideline inception | Planned coalition | Not yet started | A successful collaboration had been established between the two organisations for the COVID-19 guidelines and it was proposed that this collaboration continue with the Type 2 diabetes medicines guideline. The relationship will involve co-contribution and shared access to share data sets to reduce duplication of effort on living guidelines for type 2 diabetes medicines guideline and aid efficiencies for guideline developers working on each of these living guidelines. This collaboration is at the early stages, with a joint protocol being agreed, including what data analysis will be shared, screening and extraction guides. This likely to involve additional international collaborators. Communication is via email and videoconference. |
| Australian Living Guidelines for Diabetes (Australia Living Evidence Consortium) |
| Treatments for people with COVID-19 | National Clinical Evidence Taskforce (Australia Living Evidence Consortium) | Organisations joined collaboration during guideline development | Coalition | 1–2 years | These two organisations understood that they were both working on developing living COVID-19 guidelines, so they connected via known colleagues and arranged a meeting to discuss areas for collaboration. The collaboration was less about being involved directly in co-developing guidelines but about horizon scanning of new developments, sharing of existing search approaches, data analyses, GRADE evidence profiles, risk of bias assessments and discussing methodological approaches, panel meeting outcomes, and other aspects of developing recommendations. The coordination and communication was primarily through emails and chat platforms between the programme management and systematic review staff. Virtual meetings were held monthly throughout the collaboration to reflect on previous sharing activities and consider future work priorities. |
| National Institute for Health and Care Excellence (NICE) |
| Treatments for people with COVID-19 | National Clinical Evidence Taskforce (Australia Living Evidence Consortium) | Organisations joined collaboration during guideline development | Coalition | 1–2 years | The purpose of the collaboration was to share/adapt search strategies and literature findings, meta-analysis, risk of bias assessments and GRADE evidence profiles regarding treatment for patients with covid-19. The coordination and communication were primarily through mails and Slack between information specialist and methodologist. Virtual meetings were only set up in the beginning of the collaboration. Mails and communication on Slack was scheduled as needed – in some periods weekly or monthly. The collaboration provided the opportunity to update or develop recommendations as soon as new relevant evidence became available – hence making the process faster and providing up-to-date evidence-based guidance. |
| Danish Health Authority |
| Stroke | Knowledge Institute of the Dutch Association of Medical Specialists | Organisations joined collaboration during guideline development | Stopped early and planning to restart as a partnership or full collaboration | Planning to restart – intends to be 1–2 years | The collaboration was identified via mutual contacts and initial planning and mapping was done. However, due to misalignments in timing, different working methods other priorities and the COVID-19 pandemic, the collaboration was stopped early. The collaboration is planning to restart. |
| Stroke Foundation (Australia Living Evidence Consortium) |
| Coalition - A collective of individuals and/or organizations that has formed to work on a shared goal or product and is made up of representatives from different groups, organizations, or professions. \ |
| Shared ownership - An agreement that intellectual property and copyright relating to a piece of work or guideline belongs to more than one organization or group. |
| GRADE: Grading of Recommendations Assessment, Development and Evaluation. |
| EtD: Evidence to Decision |
| a) Canadian Agency for Drugs and Technologies in Health, Public Health Association of Canada, and Agency for Healthcare Research and Quality |
Table 1: Living guidelines collaborations (at end of document).
Twelve representatives from the participating organisations participated in the survey. Nine participants were involved in ongoing collaborations, and therefore completed the full survey. Three participants responded to a part of the survey due to their collaboration having ceased early or just starting. Of the six collaborations, one collaboration had ceased early because of evolving misalignment in the timing of work between organisations, the other collaborations were ongoing, and one was just beginning. The approaches to collaboration were established informally and through previously known collaborators. The most common length of these collaborations to date was 1–2 years (three collaborations) and two had been in place for over 2 years. The guidelines in the collaborations all involved guidelines that were living from inception, except for the living guidelines by the Stoke Foundation, which were a traditional guideline that transitioned to living.
All twelve participants stated the purpose of their collaboration was for general sharing of experiences at the topic or guideline level (100%) and eleven collaborators were working on the same specific clinical questions (92%). Five collaborators wished to develop guidance quicker (42%) and four collaborators wanted to utilise experience of other groups, such as health economics or methods expertise (33%). Only one collaboration aimed to put forward a joint position on a topic and to coordinate development of specific clinical questions. Some of the collaborations did not aim to share work but consisted more of informal frequent meetings to discuss emerging clinical priorities, and methods and processes barriers and how they were overcome. For example, the collaboration between NCET and NICE on treatments for COVID-19, did not aim to co-develop guidelines but to share existing data analyses and discuss methodological approaches, panel meeting outcomes and other aspects of developing recommendations. Similarly, the collaboration with the Danish Health Authority (DHA) and NCET, the DHA mainly adapted the NCET guideline on COVID-19.
The consolidated results of the open-ended and closed-ended questions resulted in the following themes: 1) facilitators of successful living collaboration, 2) barriers to successful living collaboration, 3) living influence on collaboration, 4) deciding to collaborate, 5) establishing mechanisms of collaboration, 6) communication and coordination of the collaboration, 7) sharing information, 8) publication, authorship, and recognition, and 9) evaluation.
Facilitators to successful living collaboration
Content analysis of the open-ended question on “What do you think successful collaboration looks like?” identified several key elements necessary for successful collaboration for developing living guidelines (Fig. 1). A mutually beneficial partnership was one element expressed by four participants.
“…a mutually beneficial partnership where the outputs are greater than the sum of the parts”.
Other elements identified included a flexible, adaptable partnership, built on high levels of trust, where each organisation is making meaningful contributions, which results in reducing unnecessary duplication of effort, minimisation of waste, increased productivity, and efficiency savings to achieve the goals of both organisations.
“Both parties understanding the desired outcome and steps needed to reach their goals, and willingness to be adaptable and work together towards achieving those goals.”
Collaboration is hard! It needs real commitment and mutual benefit for both parties.
Figure 1: Facilitators of a successful collaboration for developing living guidelines
An important facilitator was a high level of trust in the collaborating organisation’s methods and processes, and in the people in the collaborating organisation.
Shared trust in each other’s methods and standards.
Equally important was senior leadership support that the collaboration is an appropriate investment of staff time, again this often occurred when there were already established collaborations between the leaders of each organisation.
Barriers to successful collaboration
We explored barriers to collaboration via both open- and closed-ended questions (Table 2).
The biggest barrier to collaboration, mentioned by seven participants (58%), was the time zone differences between the organisations, noting that all of the collaborations were between an Australian organisation, and one or more organisations based in Europe or North America. Another barrier was the competing demands and differing time pressures and differences in resourcing and funding. Differing methodological approaches also proved challenging.
The different approaches taken for baseline characteristics and risk of bias meant we had to do some additional work after receiving the data.
Table 2
Enablers and barriers to successful collaboration
| Facilitators | Barriers |
| Senior leadership support. | Difference in time zones. |
| High level of trust in the organisation’s methods and processes, and in the people of the other organisation. | Differing priorities and demands. |
| Desire to work together to decrease duplication. | Lack of timely access to data. |
| A willingness and ability to work adaptively and flexibly without a formal structure. | Differing organisational structures and processes. |
| Willingness and ability to share information freely. | Differing data formats and issues with data sharing requirements. |
| Similar methodological approaches. | Different methodological approaches. |
| Mutually beneficial to each organisation’s goals. | Different timelines and deadlines |
| Meaningful contributions can be made by both organisations. | Lack of dedicated project coordination support for the collaboration. |
Living influence on collaboration
Eleven participants responded to the second survey to ask how the ‘livingness’ of the collaboration influenced its success (i.e. the flexibility of using a living approach to the guidelines allowed for a collaboration to evolve as per the needs of the collaborators). Ten of the participants agreed that the ‘livingness’ of the collaboration influenced its success, with only one respondent disagreeing, stating they believe collaborations can be successful regardless of whether they are living. Those who agreed noted that due to the longer-term, ongoing nature of living, there may be more investment in the collaboration and ability to evolve coordination of timings so that all participants could benefit from the collaboration.
All groups see it as a longer term investment; therefore willing to put more resources towards it
It helps with maintaining the momentum of the collaboration, and a mutual drive to facilitate it.
It also may aid collaboration via the ability to remain flexible and adjust over time.
By enabling the collaboration to evolve and respond to the needs of the collaborators as the context or evidence changes.
Aids the ability to adjust easily as things change over time
Deciding to collaborate
The primary reason for deciding to collaborate was to reduce unnecessary duplication and waste (median rating of 9/10, IQR 1.25), senior leadership support for the collaboration (median rating of 7/10, IQR 2), and utilising skillsets from other groups (e.g. qualitative or health economic analyses) (median rating of 7/10, IQR 2.5). Other factors were less commonly taken into account when deciding to collaborate, such as facilitating consistency in guidance across organisations (median rating of 5.5/10, IQR 3.25), perceived value for time or a business case that shows collaboration will reduce staff costs (median rating of 6.5/10, IQR 4.25), and potential for sharing learning and collaborating on topic areas or on evolving living guideline methods (noted in free-text by one collaborator).
Nine respondents specified that the collaboration was established via pre-existing connections or mutual contacts (60%). Two collaborating organisations were identified through G-I-N and one through a professional society. Open-ended answers also noted that some respondents contacted each other’s organisations because of awareness of the other organisation developing guidance, for example from seeing the published guideline.
Establishing mechanisms of collaboration
We asked closed-ended questions on whether the collaborations used formal mechanisms to establish the collaboration such as, conflict of interest policies, memoranda of understanding, letters of agreement, and licensing agreements [10, 11]. However, none of our collaborations used formal mechanisms to establish their collaboration. We analysed responses to the open-ended question “why were there no formal rules in place?” and found a willingness to work together without a formal structure was an emerging theme.
Willingness of both parties to share information and work together without a tight formal structure.
Following discussion with our collaborators, it was deemed that formal rules were not required so that we could remain agile and avoid potential roadblocks.
The use of formal mechanisms was deemed not necessary, due to the desire for the collaborations to be flexible and evolve over time. Others did not feel they were required to establish formal mechanisms due to pre-existing work relationships or being covered under pre-existing policies, such as those for Cochrane reviews.
Communication and coordination of the collaboration
Virtual meetings were also held by 10 collaborators (83%) between the guideline methodologists and evidence teams from each organisation, and six collaborators (50%) stated that senior leadership also may have virtual meetings. Coordination of parts of the guideline development throughout the collaboration was most common in surveillance and searching for the evidence, with eight collaborators coordinating these parts (66%). Four collaborators coordinated timelines and work planning stages (33%) and five collaborators coordinated elements of the population, intervention, comparator, outcome (PICO) (42%). Only one collaboration attended each other’s panel meetings and allocated clinical questions to each other. Although less common, three collaborators noted they changed the timing of guideline questions to fit with the other organisation, changed their methods to fit with the other organisation (25%), three collaborators coordinated the timing of updates (25%) and three collaborators coordinated clinical questions (25% collaborators).
Sharing information
Various types of information were shared during the collaborations. Eleven collaborators shared their search results (92%), nine collaborators shared search strategies (75%) and evidence summaries (75%) and eight collaborators shared data extractions (67%). Six collaborators also shared details of PICO questions, four shared GRADE evidence profiles (33%), six shared meta-analyses (50%), with three sharing network meta-analyses (25%). For the living network meta-analysis (LNMA) collaboration between ANZMUSC and CRA, the LNMA was done through Cochrane and the organisations planned the PICO together and shared all elements of the LNMA, once the LNMA was complete, then they shared the GRADE evidence profiles and EtD and adapted these to their local context.
Information was shared via email by all collaborators (100%), but chat platforms were used by four collaborators (33%), for example between DHA and NCET, who used the chat platform Slack®[31]. MAGICapp® and Review Manager™ files were shared by four collaborators. For example, if the NCCET had reviewed a particular COVID-19 treatment and NICE were planning on developing a recommendation for that treatment, they would share the Review Manager™ file and data analyses on an as needed basis. Two collaborators also used an online file sharing platform.
Only one collaborator noted challenges in institutional approaches to data sharing and sharing data through compatible file types (8%). Four collaborators’ experienced challenges in institutional approaches to format of data (33%). For example, as part of the collaboration between NICE and the Australian Living Guidelines for Diabetes, the groups are exploring ways to share data in an efficient way, and minimise the impact of internal systems leading to constraints. None of the collaborators experienced any challenges with confidentiality when sharing data.
All data is secondary use of published data, so no issues (with confidentiality).
Although less common, three collaborators (25%) changed IT programs to facilitate better sharing between organisations, changed the timing of when they reviewed the questions to fit with the other organisation, or changed their methods to fit with the other organisation. Four (33%) coordinated the timing of updates and coordinated clinical questions.
Open-ended questions also revealed that data sharing should be discussed early on in the collaboration, and there could be an agreement on terms.
Discuss data sharing requirements from both organisations at an earlier stage.
Perhaps rather than have an ad hoc sharing approach, do more forward planning to determine where collaboration could be useful and be clear on what data and information we would need.
Publication authorship, and recognition
Seven collaborators reported that collaboration did not influence publication of the individual organisation’s guidelines. Two collaborators changed timing of the publication due to the collaboration. The collaboration between ANZMUSC and CRA conducted the LNMA together and shared authorship of the LNMA as per Cochrane authorship guidelines, although still published separate local guidelines, with formal acknowledgment of other organisation in respective guideline publications[32].
Only one of the collaborations intended to develop official processes to handle any potential issues regarding authorship or ownership and another signed confidentiality agreements but none of the other collaborations had any official authorship policies.
“Shared authorship on papers; acknowledgement within the guideline.”
An example of an informal authorship process is between NICE and NCET on their COVID-19 guideline. NICE have credited NCET in their guideline methods section as certain evidence profiles and information are based on those developed by NCET, adding additional studies, or changing the analysis as needed to match NICE’s protocol. DHA also follow a similar approach for crediting work, stating in their guidelines “all recommendations in this national clinical guideline have been prepared on the basis of the evidence from the existing Australian guidelines for the critical care of people with COVID-19" prepared by the Australian National COVID-19 Clinical Evidence Taskforce.”[27]
Many collaborators felt formal mechanisms for crediting or publication authorship were not necessary due to prior shared work histories with other collaborators, the dynamic and informal nature of the collaborative relationship and the purpose of the collaboration being a general sharing rather than co-developing. One collaboration was covered by pre-existing confidentiality agreements formally signed through Cochrane.
Collaborators stated that for future collaborations they would consider official authorship policies, depending on the nature and scope of the collaboration. If the guideline is jointly produced then it would be important to establish the guidelines ownership before beginning the process and develop clear processes for development, maintenance, and updates in terms of who is responsible.
Not for this particular collaboration due to our prior shared work history but this would be potentially important for other future collaborations.
Evaluation
None of our collaborations conducted any formal evaluations of the collaborations as all are still ongoing and for some, the informality of the collaboration does not quite fit the template for a formal evaluation. The collaborators agreed that at least some informal means of evaluation could be helpful to provide clarity for collaborators.
I think we could perform more formal evaluation of our progress and continually re-set shared goals.
When asked what collaborators “would do differently next time?”, many felt the collaboration should have started earlier.
Talk earlier in the process.
Taken steps to initiate the collaboration even sooner!