Big data offer unrealized potential for impact evaluations
Traditional data collection can be costly; target populations may be inaccessible, phenomena cannot always be directly observed and interviewing people may be unethical, dangerous or impossible. In addition, budget constraints can limit data collection.
‘Big data’ does not require data collection in the field, and can provide insight into economic, social, and political behaviour.
The EGM is co-published with 3ie: https://gapmaps.3ieimpact.org/evidence-maps/big-data-systematic-map
What is this evidence and gap map about?
Big data consist of data such as online searches, social media, citizen reporting or crowdsourced data, process-mediated data such as mobile phone call record details (CRD), commercial transactions data and machine-generated data from satellites, sensors or drones.
Big data can measure outcomes that could not previously be measured using household surveys. The potential of big data to answer causal attribution, however, is still not widely understood, especially in low- and middle-income countries.
What is the aim of this evidence and gap map (EGM)?
This EGM of studies using big data reviews methodological, ethical and practical constraints relating to the use of big data.
What studies are included?
The map includes impact evaluations (IEs) that use big data to evaluate development outcomes, systematic reviews of big data IEs and other measurement studies that innovatively use big data to measure and validate any development outcomes.
Sources include social networks, internet searches, mobile data, crowd sourced data, data from public agencies, data produced by business, call record details, and satellite data.
The map contains 437 studies written in English published between 2005 and 2019: 48 IEs, 381 measurement studies and eight systematic reviews.
What are the main findings of this map?
There is considerable potential for measuring various development indicators using big data. The measurement studies serve as a proof-of-concept for evaluators wanting innovative ways to measure development outcomes.
There is potential for more IEs on development interventions. The map shows that the number of IEs that use big data to measure outcomes or control variables is growing fast, and there is scope for greater use.
Satellite data is used the most. The use of satellite data for IEs and measurement studies has been facilitated by the availability of pre-processed satellite data, new machine learning techniques and increased computational capacity to process the satellite images into meaningful measures of development outcomes. Despite a number of high-profile measurement studies, CRD data has not been used to rigorously evaluate any development outcome. Similarly, other human-sourced data and process-mediated data have been used only sparingly in IEs.
There are potential sectors and themes where systematic reviews will be useful. The map highlights a few potential thematic areas where systematic reviews will be informative, most notably those of (i) all IEs that have used satellite data; and (ii) those with reference to the data sources used in rigorously evaluating forest cover.
The number of measurement studies indicates potential for more IEs in fragile contexts.
Ethical concerns and transparency issues are substantial. Ethical issues related to informed consent, data privacy, data security and unintended exclusion are severe for some of the sources of big data. Few studies report on ethical issues related to using big data.
Some capacity constraints are acute. Computational capacity is constrained and technical expertise on large-scale big data analysis is siloed.
What do the findings of the map mean?
This map shows that big data can contribute to the evidence base in development sectors where evaluations are often infeasible due to data issues.
One of the key ‘absolute gaps’ that the map has identified is that there are fewer IEs than measurement studies. Given the fast-growing availability of big data and improving computation capacity, there is great potential for the use of big data in future IEs. However, analytical, ethical and logistical challenges may hinder the use of big data in evaluations.
This report calls for standards to be set for the reporting of data quality issues, data representativeness and data transparency, and an Institutional Review Board review specifically designed for ethical issues related to big data.
More interaction is needed between big data analysts, remote sensing scientists and evaluators.
How up-to-date is this EGM?
The authors searched for studies published up to December 2019.