This draft is to seek feedback from across the collaboration and the wider evidence synthesis community as the Campbell Collaboration prepares our 2024 plans.
The need
It should be as natural to find out the state of the art in evidence about how to tackle the world’s most important social challenges as it is to look up a word in a dictionary. And it can be.
Evidence synthesis finds out what works to tackle social challenges by testing and synthesizing all the relevant research. It produces a body of knowledge from lots of bits of knowledge. It helps improve outcomes, find efficiency savings, and reduce waste.
Evidence synthesis can be more useful, faster, and permanently cheaper. The ingredients for achieving this are largely already known. The task is to bring them into routine use in a way that benefits everyone.
This document sets out five commitments to achieve a step change in evidence synthesis capability globally. A step change in evidence synthesis requires commitments from across the evidence ecosystem. Evidence synthesis producers; evidence producers; funders and commissioners and evidence synthesis technology producers working together can achieve lasting benefits for everyone, across the world and across disciplines.
The Campbell Collaboration seeks to play an enabling role in that change, in the service of the wider global evidence community. So the Campbell Collaboration will make these commitments in our own work and work on an implementation plan to start to fulfil them in 2024 and beyond. Our goal is to develop an implementation plan collaboratively with partners across the global evidence synthesis and wider evidence communities so we are sharing this thinking in early draft form to invite anyone who shares these commitments to say so publicly and to work with us to fulfil them.
Five commitments
- User focus: deliver decision-ready synthesis
- Sharing and reusability: open science including publication, data, tech, and education
- Effective AI: move tech from experiment through quality assurance and productization to deployment
- Supporting people to do their best work: embrace specialization and cross-functional teams, and support learning and development
- Methods and process innovation: ensure our processes and guidelines support teams to make the most of the full range of evidence
1. User focus
We do evidence synthesis to contribute to a better and fairer world. Enhancing how evidence synthesis is used is central to our work. We serve our users best by demonstrably maintaining high standards of quality. We are committed to working in the service of four types of users:
- Decision makers of all kinds, including the public: by working with diverse decision makers to define and then equip ourselves to deliver decision-ready evidence synthesis products that are adaptable to support decision-making in different contexts. This can include timely interim products clearly distinguished from final products.
- Synthesis funders and commissioners: by supporting expert commissioning by working with partners to develop agreed open educational resources for anyone who might benefit from commissioning evidence synthesis, both ensuring high quality answers to high quality questions for commissioners, and an institutionalized demand-side pressure for high quality from producers (where there are concerns that sometimes the quality / cost tradeoff goes wrong in procurement).
- Research funders and commissioners: by helping synthesis become a driver of research quality and value for money by ensuring we use synthesis to show the kinds of research that will and will not make a difference for decision makers.
- Other researchers: by committing to sharing and reusability as set out below.
2. Sharing and reusability
We support the UNESCO Recommendation on Open Science (https://www.unesco.org/en/open-science/) and endorse the need for “reproducibility, transparency, sharing and collaboration resulting from the increased opening of scientific contents, tools and processes.”
We commit to working in the open with others on four major aspects of open evidence synthesis:
- Open publications: enabling open access, unrestricted distribution, interoperability and long-term digital preservation and archiving.
- Open data: from each stage of the synthesis process shared so that it can be openly used, reused, retained and redistributed by anyone, subject to acknowledgement, in user-friendly, human- and machine-readable and actionable formats according to FAIR principles of Findability, Accessibility, Interoperability, and Reusability (see https://www.go-fair.org/).
- Open source: when open source code is a component of a research process, enabling reuse and replication generally requires that it be accompanied with open data and open specifications of the environment required to compile and run it.
- Open educational resources: teaching, learning and research materials in any medium – digital or otherwise – that reside in the public domain or have been released under an open licence that permits no-cost access, use, adaptation and redistribution by others with no or limited restrictions.
Evidence synthesis depends on accessing knowledge from primary research and other sources. We will work with producers, commissioners and others involved in primary research to support open publication and FAIR open data approaches that enable modern faster cheaper synthesis production.
3. Effective AI
AI offers opportunities for many aspects of the evidence synthesis process.
The most immediate and biggest win is in speeding up and reducing the human workload in the search process. This has been developed and productized but not yet reached user acceptance. The International Collaboration for the Automation of Systematic Reviews has called for formal evaluation of tools in this area by leading evidence synthesis organizations so they can be incorporated into editorial processes when and if they pass evaluation. This work can be done quickly if backed, as we intend to.
Other areas such as data extraction where big benefits are also possible are still at the development stage.
Effective automation is not just about hard AI. Relatively simple workflow tools such as templates and tools to translate search syntax between different databases can have big benefits.
The stages of this work can be summed up as:
- Development: needs meaningful funding (for open source), appropriate teams, close collaboration between technical and domain experts. In machine learning, as in synthesis, task definition is a vital stage, and then getting appropriate training data sets is crucial. Again it is vital that training and evaluation data is an open resource.
- Evaluation: needs leadership from evidence synthesis organizations. We and partners are in the early stages of putting together a joint project to get this going and plan to launch an open invitation for participation together in the coming weeks. Evaluation creates the knowledge needed for either further development or deployment with appropriate training and guidelines.
- Productization: incorporating technical capabilities into usable products. Two of the most used synthesis tools, EPPI Reviewer and Covidence, are already committed to supporting automation functionality, which is an important enabler.
- Deployment and user acceptance: requires (a) functioning tech (b) proof that it is functioning appropriately (c) the tech embodied in usable products (d) agreed guidelines for appropriate use (e) training (f) ongoing support.
4. Supporting people to do their best work
Evidence synthesis projects often work as one end-to-end process where one centralized team works on everything from defining the question to searches, coding and data extraction, statistical analysis, and eventually dissemination and user engagement. In that context, academic and research skills tend to be treated as more essential than user engagement skills, and longer-term innovation such as using AI tends to depend on individual teams’ enthusiasms and is therefore progressing more slowly than the world needs.
In a larger-scale synthesis project there is an opportunity to build a team where different specialists come together from across a global community in mutually supportive ways, to support specialization for individuals in different aspects of the process, and to gain economies of scale from doing that. To that end:
- We will develop an operating model for commissioning and delivering evidence synthesis that integrates different specialists working together and achieves economies of scale across parallel synthesis projects. This may need to begin with larger-scale partners who have a wider range of skills in house, or using an intermediary to broker the teams, but over time we aim to support contributors of all kinds and anywhere in the world to work in this way and to adapt it to their context.
- We will continue to develop training and accreditation to support different specialisms at entry level, early career, mid-level, and expert level, extending this to cover skills beyond research where appropriate.
- Recognizing the bottleneck of senior evidence synthesis specialists volunteering as editors and in other leadership roles, we will support the full pipeline of learning and development, enabling our leaders to develop new leaders, our contributors to step up to lead, and new contributors to come in.
- In all this, we will treat policy, communication, and technology as necessary specialisms within the evidence synthesis process alongside research specialisms.
5. Methods and process innovation
- We will work to simplify and streamline research and editorial processes, recognizing the need to support consistent high expectations in different local, disciplinary, and other contexts.
- Ensure we have guidelines for all the different kinds of synthesis we need.
- Ensure we have guidelines for all the different kinds of evidence we may wish to use in evidence synthesis (for example, working with evaluation specialists on how best to incorporate evaluative evidence).
- We will seek to work with others to merge, deduplicate, and simplify standards and guidelines where possible.
This draft is to seek feedback from across the collaboration and the wider evidence synthesis community as the Campbell Collaboration prepares our 2024 plans. Our goal is to develop an implementation plan collaboratively with partners across the global evidence synthesis and wider evidence communities.
Your feedback is welcome to Will Moy, wmoy@campbellcollaboration.org.