By Howard White, CEO
When I present my evidence pyramid, shown below, I raise a smile by mentioning the World Health Organization (WHO) “guidelines on guidelines”. Its actual name is the Handbook for Guideline Development. The handbook is the basis for WHO guidelines spanning a wide range of topics, from the quality of drinking water, and child obesity, to male circumcision and indoor air quality.
All of these guidelines – following the guidelines laid down in the guidelines on guidelines – are based on high-quality systematic reviews. Not literature reviews, not rapid reviews (guideline development takes two to three years, partly to allow for any necessary reviews to be done), and certainly not expert opinion.
The evidence pyramid
White in Gorard (2020), adapted from White (2019)
Evidence-based practice is strongest in health. Institutionalisation of the use of evidence, as shown by WHO’s approach to guideline development, is common in health agencies around the world. What about other sectors?
In Stephen Gorard’s new book Getting Evidence Into Education, I present a comparative scorecard for the global evidence architecture in health and education, with a partial focus on developing countries. The organising framework for the chapter is the evidence pyramid shown above. A brief summary is as follows:
Data: The comparability of research findings is helped when researchers use common concepts. In health, WHO maintains both the International Classification of Diseases (ICD) and the International Classification of Procedures in Medicine (ICPM). The COMET initiative has established core outcome sets to use in medical studies. Demographic and Health Surveys produce internationally comparable data on a wide range of indicators such as infant mortality, teenage pregnancy and immunization coverage in countries in which administrative data cannot be relied upon. There is no similar initiative to standardise education intervention categories or outcome measures. The only globally comparable data are from the TIMMS and PISA education outcome tests, which include few developing countries.
Studies: The evidence movement is mostly concerned with effectiveness studies, also call impact evaluations. Where possible – as it is in both health and education – randomised controlled trials (RCTs) are the best approach to assessing if an intervention works or not. A review of RCTs in education found 1,017 studies. In contrast there are over 1.5 million studies in Cochrane’s clinical trials database. And this evidence really matters, as both IES in the US and the Education Endowment Fund (EEF) in England have found that the RCTs they fund show that 80% of education interventions don’t work at improving education outcomes.
Systematic reviews: A search of Google Scholar shows there are also around 1,000 education systematic reviews. The health evidence database, Epistemonikos, has over 38,000 reviews. The Cochrane Library alone has over 8,000 reviews.
Databases and maps: Both health and education have large general research databases – PubMed (which includes Medline) for health and ERIC for education. But in health there is also Epistemonikos, exclusively devoted to studies of effectiveness, which education lacks. Epistemonikos also allows users to map the evidence. Evidence maps have grown in popularity very recently. There are various maps related to both health and education, notably a map of all education studies in low- and middle-income countries, produced by the International Initiative for Impact Evaluation.
Evidence platforms and portals: Evidence platforms present a curated collection of evidence, such as links to new studies. Some platforms, like healthevidence.org, may also include a database. A notable education platform is “Education World: connecting educators to what works”. However, from the point of view of users, platforms share the same limitation as databases and maps. They take the user back to the original research.
Many practitioners don’t read research reports. Hence the top three levels of the evidence pyramid are evidence products which allow users to make evidence-based decisions without having to read the underlying research. The first level is for evidence portals, which summarise the evidence of effectiveness of different interventions based on systematic reviews. Evidence portals are central to the What Works movement, with IES’ What Works Clearinghouse and EEF’s Teaching and Learning Toolkit being leading examples. I call the toolkit the best website in the world. Health is moving toward online evidence tools such as MAGICApp.
Guidelines and checklists: The education sector lacks guidelines and checklists at both national and global levels. UNESCO has a few, but not on anything like the scale of the WHO, and not evidence-based in the same way. In the UK, EEF has started to produce guidelines, but not with the institutionalised approach of its health counterpart, NICE.
Education is badly and sadly lagging health in the production of evidence and evidence-based products
So, overall, the scorecard shows that education is badly and sadly lagging health in the production of evidence and evidence-based products. There are some highlights near the top of the pyramid, such the evidence portals in the UK and US. But you can’t build the top of the pyramid without its base. And the base of the pyramid is much weaker in education.
Why invest in the education evidence architecture
And that is why we need a Global Education Organisation. There is no agency systematically investing in the global public good of the education evidence architecture. As a global public good, this is the job of multilateral agencies. It is precisely why they are created. However, UNESCO is under-resourced and has no track record of commissioning large amounts of effectiveness research. The Global Partnership for Education (GPE) may be different. It has literally billions of dollars to spend on ‘evidence-based’ education approaches in developing countries. Less than half of 1% of its resources would be sufficient to make a substantial difference to the global evidence architecture for education. Yet it devotes a tiny share of these resources to developing the evidence architecture and does not do so in a systematic way. That is a shame, as what evidence we have suggests that most interventions don’t work, so a good share of GPE’s billions of dollars are being spent on ineffective interventions.
A start would be to commission and maintain an evidence map, i.e., annually update the 3ie education map. And then to commission systematic reviews where they are lacking. And then to produce a global education evidence portal.
Some of this work is already ongoing, notably by IES and EEF. So, there is a need for coordination. When it comes to systematic reviews, that coordination mechanism already exists in the form of the Campbell Collaboration. Campbell is the international repository of systematic reviews in a range of sectors, including education. See a selection of our education reviews in our recent virtual issue on education.
I call on:
- practitioners to consult our reviews, all of which come with a plain language summary
- researchers conducting reviews to register them with Campbell
- commissioners to work with us to commission sets of reviews to fill the evidence gaps (and work with 3ie to fill evidence gaps for primary studies)
- and the global donor community to get their act together in strategic investment in the evidence architecture for education.
Viva la revolución de la evidencia!