Here be monsters? Innovations in evidence mapping

By Howard White and Ashrita Saran 

In the concluding session of last year’s Global Evidence and Implementation Summit one participant commented that this summit has been about maps, maps, maps.

There has indeed been a huge rise in evidence mapping. A map of maps in international development led by 3ie as part of the Centre of Excellence for Development Impact and Learning found 79 maps, most of which were commissioned within the last couple of years.

Evidence mapping is not new. Our review of evidence mapping found the first maps in the early 2000s. But the current wave of mapping was given a boost by the work on 3ie on its first map in 2010. 3ie developed the now familiar pictorial representation of evidence and gap maps (EGMs) as a matrix with interventions in rows and outcomes in columns, which are interactive so the user can access the evidence in the map. Examples of 3ie EGMs include forest conservation, peacebuilding, and intimate partner violence. To date 3ie has over 20 completed on or ongoing evidence maps. You can read more about it in the award winning paper of Birte Snilstveit and other 3ie staff in the Journal of Clinical Epidemiology.

Maps found an eager following, with many agencies working with 3ie to produce maps, such as IRC, Sightsavers and USAID. The Campbell Collaboration was keen to transfer the success of this evidence synthesis product to other sectors. In 2018, we opened our doors to accept publication of maps.

Innovation in scope and content

So where are the monsters? 'Here be monsters' is an expression medieval cartographers are said to have put at the edge of maps beyond the known world. As we started work on maps, we went off the map of existing maps to innovate in approaches. The innovation has been both in the scope of maps and in their content.

The figure shows the range of evidence synthesis products which have varying scope. There is a trade off in scope and content, meaning depth of analysis. A systematic review reviews primary studies and can have deep contextual analysis. A review of reviews is likely to be broader in scope but have less on context, as it is limited to what it can code across the set of reviews studies. Maps contain both primary studies and reviews but are based on much more limited coding. Most importantly, maps report what evidence is there but not what is says. We also proposed mega-maps, which have a very broad thematic scope and so only include reviews and other evidence and gap maps not primary studies. Our mega-map of child welfare includes over 300 reviews. And then there is the map of maps, which has a very broad scope indeed – such as the whole of international development – and so only includes other maps.

We are also innovating with respect to content. Most maps are maps of evidence of effectiveness. But just like systematic reviewing in an approach which can be applied to any research question, the same is true of maps. So, working with the UK Centre for Homelessness Impact, we produced maps both of studies of effectiveness and of process evaluations, called the implementation issues map. In an effectiveness map the columns typically show the outcomes. But in the implementation issues we coded the barriers and facilitators identified in the over 220 included studies.

Mapping development in Uganda

In another map – our first country evidence and gap map for Uganda launched in February this year – we have combined impact evaluations, process evaluations and formative evaluations. We found around 500 evaluations of development interventions published in Uganda since 2000, most of which are little read and less used. (Read more about this here.) By putting them in the public domain we hope to increase the use of the evidence they contain.

The innovations of the type of map have required innovation in review methods. The search strategy for a country map is very different to the search strategy to the traditional database-led strategy for most effectiveness maps and reviews. We also innovate in what we code. The latest version of the homelessness effectiveness map includes critical appraisal of all included studies. We are next turning to critical appraisal of included studies. Another map, led by IMANNA, is mapping methods and metrics rather than interventions and outcomes.

To date there are 18 EGMs in the Campbell online library at various stages of production. We are taking the production of maps off the charts. Yes there are monsters – ignorance and un-evidence programmes. We fight them with evidence.

Join the evidence revolution

If you are interested in doing a map contact Campbell or 3ie so you can get the benefit of our experience. Register the map with Campbell to increase its credibility and discoverability.

Contact us