Methods and reporting of Campbell reviews are generally good: further improvement is possible
Campbell systematic reviews of research on effectiveness of interventions are, in general, of high quality in terms of meeting international standards for the conduct and reporting of systematic reviews, though improvements could be made in several aspects.
Campbell reviews published after the introduction of the checklist of methodological expectation (Methodological Expectations of Campbell Collaboration Intervention Reviews [MECCIR]) in 2014 had better reporting and methodological qualities, however, this may be due to more global trends toward improvement over time.
What is this review about?
The Campbell Collaboration undertakes systematic reviews of the effects of social and economic policies to help policymakers, practitioners, and the public to make well‐informed decisions about policy interventions. There has not been a comprehensive review of the methods and reporting characteristics of Campbell reviews. Nor has the methodological quality and completeness of reporting of Campbell reviews been assessed. These factors, which are assessed in this review, are important to ensure the transparency, reliability, and usability of Campbell systematic reviews.
What is the aim of this review?
We collected information about the epidemiological, methodological, and reporting characteristics of Campbell reviews of the effects of policies published between 2001 and 2018, and assessed their methodological quality and completeness of reporting. We also assessed whether the methodological quality and completeness of reporting were better following the release of the MECCIR standards and in reviews that were co‐registered with Cochrane.
What studies are included?
This review included 96 Campbell reviews that evaluated the effectiveness of any intervention. Forty‐six were published between January 2011 and September 2014 and 50 published between October 2014 and January 2018. Twenty‐two of the 96 reviews were co‐registered with Cochrane.
What are the main findings of this review?
Fifty‐nine percent of the Campbell reviews were of high or moderate quality based on the AMSTAR 2. Twenty‐five of 96 (26%) Campbell reviews completely or partially reported all 27 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta‐Analyses) checklist items. The median number of items reported across reviews was 25 (interquartile range [IQR], 22–26).
The methodological quality of Campbell reviews improved after the introduction of MECCIR, which might reflect a trend of improving quality over time. Seventy‐six percent of reviews published after the introduction of MECCIR had high or moderate methodological quality (AMSTAR 2) compared with 39% of reviews published before MECCIR.
Campbell reviews co‐registered with Cochrane had better methodological quality compared with those only registered on Campbell. Seventy‐seven percent of reviews registered in both Campbell and Cochrane were of high or moderate quality compared with 53% of reviews only registered with Campbell.
What do the findings of this review mean?
Campbell reviews are generally well reported: 59% of the included reviews are of moderate to high methodological quality, according to PRISMA and AMSTAR 2 standards for systematic reviews. Campbell reviews published after MECCIR were of better quality than pre‐MECCIR, and reviews co‐registered in Cochrane scored better on quality indices than reviews only registered in Campbell.
This report may be used as a baseline for assessing effects of implementing strategies to address limitations identified by this review (such as interpreting risk of bias, reporting funding of individual studies, explaining the selection of study designs, citing the pre‐published protocol as well as other items). Future studies may be useful to explore more detail on specific methods such as search strategies and economic considerations.
How up‐to‐date is this review?
This review included all the Campbell effectiveness reviews published between January 2011 and January 2018.