ESMARConf as an active community of practice working on free and Open Source evidence synthesis methods and tools
The recent pandemic forced many research community activities and conferences online. Here, we report on a new online conference for evidence synthesis and meta-analysis in the R programming environment (ESMARConf), which focuses on accessibility and equity as core missions. ESMARConf is an event series geared towards a novel community of practice of tool developers and users working with evidence synthesis and meta-analysis in R. We believe this conference has the power to curate a vital community of practice across disciplines, raising awareness, fostering capacity sharing, and developing free and Open Source, efficient and rigorous needs-based tools to support evidence synthesis. ESMARConf builds on the success and groundwork of the broader R community and systematic review coordinating bodies (e.g. Cochrane), but fills an important niche; catering to a variety of audience types including developers, users and learners. Here, we introduce ESMARConf and its core values, calling for the evidence synthesis and R communities to fully embrace both the conference and this style of event in an effort to maximise accessibility and equity.
The need for rigorous evidence synthesis methodology
Rigorous evidence is vital across disciplines for efficient, appropriate, and fit-for-purpose decision-making with minimised risk of unintended harm, and to make the best use of available resources (1). Evidence synthesis is a broad term to describe the process of bringing together information from a range of information sources, that include rigorous systematic reviews. It is the most robust means of summarising bodies of evidence in a way that minimises bias and maximises comprehensiveness, accuracy, repeatability and reliability (2-4). In several disciplines, evidence synthesis ecosystems (5-7) have been developed that aim to ensure evidence-informed decision making is built on a solid foundation of rigorous systematic reviews: for example a planned project aims to develop a community for Open Research Synthesis in preclinical research evidence (8).
To date, however, disciplines have been slow to share evidence synthesis frameworks, best practices, and tools amongst one another. Instead, evidence synthesis paradigms are borrowed but largely reinvented for new disciplines wishing to learn from existing systems in other fields. This exceptionalism and limited collaboration undoubtedly results in inefficiencies and unnecessary teething problems for disciplines starting out and those wishing to advance evidence synthesis (9, 10). However, some efforts have been made to support interdisciplinarity in evidence synthesis: for example, in 2015 the Global Evidence Synthesis Initiative (GESI) brought together a network of centres working on evidence synthesis and knowledge translation across fields in low- and middle- income countries.
The benefit of free and Open Source software environments
The importance of communities of practice have also been recognised for some time in existing interdisciplinary contexts. Developments in collaborative digital and programmatic frameworks, such as the free and Open Source software R, have significantly expanded the opportunities for development of free-to-use, incrementally improvable, community driven tools to support evidence synthesis. Examples of such R-based evidence synthesis tools include: EviAtlas for visualising cartographic evidence maps (11); robvis for visualising risk-of-bias assessments (12); PRISMA2020 for visualising PRISMA-compliant flow diagrams (13); metadat for accessing meta-analytic datasets for training and testing purposes (14). Additionally, by being Open Source, these tools are an important component of ensuring the evidence synthesis process is computationally reproducible (15-18).
Although the community of R tool (i.e. package) developers is active and strong across disciplines, evidence synthesis (and meta-analysis) practitioners and methodologists who make use of R remain relatively disconnected from one another, often existing outside and potentially unaware of systematic review coordinating bodies, such as Cochrane, the Campbell Collaboration and the Collaboration for Environmental Evidence. Furthermore, the R community of tool developers is not well connected with users of evidence synthesis tools. This means that many needed tools may go undeveloped, and tools that could have a broad applicability may not end up being as widely used because end users are not aware of their existence or because certain use cases and needs may not be met.
On the need for an evidence synthesis technology community of practice
Clearly, then, there is a need to develop a community of practice of evidence synthesis tool developers and users. Such an interdisciplinary community that is active and well connected would facilitate the production of fit-for-purpose, needs-driven tools that conformed to rigorous standards in evidence synthesis best practice. A community would also facilitate capacity sharing and awareness raising across the use of evidence synthesis tools and methods. Furthermore, as some have argued (e.g. 19), this also leads to a more creative ecosystem that develops synergistically to meet needs and to maximise efficient use of resources. In short - a more reliable, efficient, and flexible evidence ecosystem.
The development of ESMARConf
In 2020, active members of the Evidence Synthesis Hackathon (https://www.eshackathon.org/), an organisation that aims to produce Open Source evidence synthesis tools, initiated the first Evidence Synthesis and Meta-Analysis in R Conference (ESMARConf; https://esmarconf.github.io/), held on 21st and 22nd January 2021. The event consisted of 39 presentations of 7 minutes, 10 panel discussions, and 4 workshops. During the days of the conference (21-22/01/21), the conference received 514 registrations, with 650 unique viewers during the conference and >3,500 video views.
In February 2022, our second event, ESMARConf2022, consisted of 28 presentations, 3 panel discussions, and 6 workshops. The event received 843 registrations, and the content has been viewed by 1,120 unique viewers, corresponding to 4,100 views (as of March 7th 2022).
ESMARConf is a purely online conference, designed to be as accessible and equitable as possible (see Figure 1), whilst delivering state-of-the-art and impactful content to build and share capacity, work collaboratively, and develop needs-based tools.
Importantly, we aim to combine presentations of existing tools and frameworks in R with capacity building and training both around the use of R in evidence synthesis and rigorous evidence synthesis methodology itself. By leveraging R as an open and widely used platform for statistics and data visualisation, we have the opportunity to connect a large and disciplinarily diverse audience of graduate students and earlier stage researchers with best practices in rigorous evidence synthesis. Indeed, our post-event evaluations demonstrate the importance of these career stages, but also demonstrate continued engagement from more senior researchers (Figure 2).
We developed a mission for ESMARConf that builds on the principles and values of R both as a coding platform and a community of practice, and the communities leading and promoting rigorous systematic reviews (Cochrane, Campbell, and CEE).
We aim to make ESMARConf as accessible and equitable as possible through the following core values and initiatives:
- Conference fees should not be a barrier to participation. High conference fees are a huge financial barrier to those who most need access to information and networking. We believe that online events should not be costed at the same rate as traditional, in person conferences. Some of these high-cost events undoubtedly charge what they can rather than what they cost. In fact, we believe that running online events using free and low-cost software (such as Google Forms, YouTube and Zoom) is vital for enabling low/no registration fees and facilitating equitable participation. We now have the experience of running two large, global conferences entirely for free for participants. We aim to ensure ESMARConf is free forever, but there are costs to ensuring these events are accessible (e.g., time to organise and coordinate events, translation/subtitling services) and so we welcome voluntary donations and registrations where participants’ budgets allow. We are also striving to develop Open Source automated pipelines for ESMARConf to minimise burden on conference organisers, and we welcome others who may wish to use them for the same purposes (see the draft ESMARConf_updater repository; https://github.com/ESHackathon/esmarconf_updater).
- Online conferences remove barriers to physical attendance. By holding the conference online and recording all live content, participants can digest the content at a time that works for them. Around 67% of the more than 9,500 views of the ESMARConf2021 videos on our YouTube Channel were for individual pre-recorded talks, and 73% of all views occurred after the end of the conference (see Figure 3). The vast majority of engagement with the conference happened in ‘catch-up’ rather than live, highlighting the benefit of providing for participants on their own time. Furthermore, by allowing presenters to pre-record their talks and play these during the livestream, we reduce the need for physical presence and allow presenters from any timezone to take part: 73% of presenters stated that pre-recording was extremely or moderately good and no respondents indicated that having to pre-record their talk was bad (Figure 4).
- All participants should be equally welcome and valued. Accessibility and equity are vital cornerstones to the event and our community of practice. We believe that people have an equal right to take part in the conference in a safe and welcoming environment. Our Code of Conduct and complaints and feedback procedures (https://www.eshackathon.org/about/accessibility_and_codeofconduct.html) are continually evolving and central to all our events. We mention them at the start of each live session and clearly link to them on our website. Some 83% of participants responding to an evaluation of ESMARConf2022 said that they felt personally welcome (see Figure 5).
- Access should not be restricted by the ability to hear. Along with attempting to ensure participants can join the conference according to their own schedule, we believe that participation should not rely on an ability to hear. To that extent, we ensure that all pre-recorded individual talks are subtitled, and that the transcripts are verified in English before the conference begins. We could not do this without the wonderful dedication of our presenters, who ensured recordings were sent in on time to allow for subtitles to be produced and checked manually, and without generous funding from Code for Science & Society to fund a team of transcribers and editors.
- Access should not be restricted by language. Similarly, we do not believe that language should be a barrier to taking part in ESMARConf. The verified subtitling described above allows users to automatically translate closed captions into any language provided by YouTube (currently more than 155). In future years, we hope to be able to allow presenters to present in the language of their choice in pre-recorded talks, providing subtitles that can be translated into the viewer’s preferred language.
- Access should not be restricted by geographical location. By running the event as an exclusively online conference, we aim to allow anyone with access to an internet connection (including mobile or satellite internet) and a charged/mains powered electronic device that can display a YouTube video to join. In particular, we aim to facilitate participation from people in traditionally underrepresented regions and countries. ESMARConf2022 was attended by participants from across 86 countries (see Figure 6). With a substantial number of viewers (173 of 1,120) coming from low- and upper- middle income countries; something we aim to increase in the future. In general, we believe this substantially reduces barriers otherwise present for face-to-face conferences.
- Participation by caregivers should be encouraged and supported. Many conferences create barriers to participation by caregivers, including parents and those supporting people living with disabilities. By being online, we hope to reduce some of those barriers. However, we understand that finding time to join an online conference for multiple days is still a substantial challenge for participants with caregiving responsibilities. To support our caregiving participants, at ESMARConf2022 we provided bursaries of up to 100 USD: this funding could be put towards a wide variety of costs, including alternative caregivers, quiet space hire, or meal preparation, just to name a few.
- Resource constraints should not prevent participation. We understand that many potential participants face resource constraints when trying to join an online conference, particularly those in low- and middle- income countries. To that end, at ESMARConf2022, we provided bursaries of up to 100 USD to be used to overcome resource constraints. These bursaries were used, for example, to cover the cost of commuting to a town with strong internet connection, hiring a quiet space or a location with stable electricity, hiring a diesel generator and mobile internet, purchasing headphones, etc. In total, 26 bursaries were awarded across both caregiver and resource constraint bursary types.
Sustainability Challenges and Call to Action
There are clear limitations and challenges behind organising such an event. Firstly, it relies on the efforts of a voluntary organising group. From 2021 onwards, modest funding has been obtained that includes a budget line for salary costs for key organisers working to coordinate the event itself. But a broader group of volunteers to make key decisions, help advertise the conference, and facilitate sessions is still needed. Furthermore, it also relies on a sustained interest from presenters and workshop providers contributing their time for free. Funding is also needed to cover the costs of verification of subtitles, provision of bursaries, and small software costs (namely, a Zoom subscription to support live streaming to YouTube). However, these costs are not excessive for a conference that can easily be scaled up in participation without additional costs. In our evaluation of ESMARConf2022, 75% of presenters and 37% of participants supported the idea of a voluntary registration fee to support attendance by those with accessibility and resource constraints. For participants, the median voluntary registration fee people were willing to pay was 50 USD. This indicates that an annual budget of c. 15,000 USD might be possible (although the willingness-to-pay was likely higher in those choosing to respond to the evaluation). This is a promising indication that an objective to break even in the long term might be feasible, considering that our costs for ESMARConf2022 were c. 7,000 USD.
ESMARConf aims to provide for three key audience groups and their objectives: coders producing evidence synthesis tools wishing to collaborate; current/potential R users wishing to conduct an evidence synthesis in R and learn more about the tools available; non-R users with expertise/experience in evidence synthesis wishing to use free tools via non-coding user interfaces. As a result, collaboration, awareness raising, capacity building and sharing, and matching users to developers are core objectives of the conference: more generally speaking, developing a community of practice in tool use and development.
We call on members of our three audiences to join the evidence synthesis and meta-analysis in R community of practice by embracing ESMARConf. Whatever your background in evidence synthesis, meta-analysis or R, you can play a vital role in ensuring that free and Open Source tools are developed that are fit-for-purpose, widely used and accessible to all. By joining and supporting the community, you can help us to raise awareness, build and share capacity and improve the rigour of evidence syntheses across disciplines. Support us by registering for ESMARConf2023 next year and engaging with presenters and colleagues, by presenting your work and sharing your skills, and by helping to develop needs-driven tools to improve accessibility, efficiency, and rigour of evidence syntheses across disciplines. Where possible, donations and voluntary registration fees can ‘pay it forwards’, helping to support those with constraints to break down barriers and participate in a global community.
By supporting ESMARConf, you can help to maximise accessibility and equity across regions, contexts, and social backgrounds, helping to aim for a level playing field in a digital, connected and online future of evidence synthesis. Welcome to ESMARConf!