For years, randomized control trials (RCTs) have been considered the ‘gold standard’ of development project impact evaluation. RCTs are field experiments designed to analyze the effects of development interventions through the use of treatment groups who participate in the intervention, and control groups who do not participate and serve as a counterfactual—demonstrating what would have happened in the absence of the program.
RCTs produce rigorous results, but they are time- and resource-intensive to design and implement. They also can pose difficult ethical questions of who gets a potential benefit (the treatment group) and who doesn’t (the control).
A new AidData Working Paper by a team of ten researchers at AidData, the World Bank, KfW Development Bank, and the Global Environment Facility demonstrates how geospatial impact evaluations (GIE) can address these challenges while still producing a rigorous analysis of impact. By fusing geolocated intervention data with in-situ and remotely-sensed outcome data, and applying quasi-experimental methods to control for confounding variables (outside influences that change the effect of an experiment), GIEs can establish and measure the impacts of development projects at the local level. Cheaper and faster than RCTs, but more rigorous than performance evaluations, GIEs are increasingly filling in the “missing middle” of development impact evaluation.
GIEs can be applied across a wide range of sectors (among them, infrastructure, health, and education) so long as precisely located project data can be used in conjunction with outcome data—thus enabling the method to serve as an impact evaluation tool for both individual projects and entire project portfolios.
Seeing with GIEs
The ability to build off the analytical soundness of RCTs, while avoiding some of their costs and limitations, is part of what makes GIEs so useful.
RCTs randomly assign individuals to either a treatment group or a control group, thereby ensuring that there are no systematic differences in individuals between the two groups. Over the course of the program—typically several years in the case of development projects—both groups are surveyed to determine how the program benefited the lives of those who participated. As the two groups studied were initially identical, any sort of welfare gain experienced by the treatment group could only have come from the program they participated in. The establishment of a clear causal link, between the program and the development gains it created, has made RCTs the evaluation gold standard.
But the features that give RCT results their rigor also increase costs. The evaluation must be in place before the program begins, and data collection—frequently expensive and on-the-ground—on both the treatment and control groups needs to occur before, during, and after the program.
Program managers must also consider moral questions surrounding the use of control groups in RCTs. If researchers have reason to believe that participation in a development program could improve lives, is it ethical to keep control subjects from participating in the program and enjoying its benefits?
By contrast, GIEs do not rely on experimental methods to separate participants into randomized treatment and control groups. As outlined by Ariel BenYishay, AidData’s Chief Economist, and co-authors in their new primer on GIE methods, tools and applications, GIEs need four key “ingredients” in order to work: (1) spatially-explicit project data, (2) spatially-explicit outcome and covariate data, (3) the ability to join these data, and (4) econometric tools that can properly analyze geospatial data. GIEs can leverage an ever-growing wealth of remotely-sensed and observational data to meet these requirements, allowing them to be conducted more quickly and cheaply than RCTs.
In place of randomization, GIEs create models that establish the counterfactual cases needed for sound statistical analysis through one of three methods outlined in BenYishay et al. These methods take advantage of available data to create statistically similar treatment and control groups. In this way, a solid counterfactual can be created without having to divide individuals or other study units into actual treatment and control groups. In addition, the ability to model treatment and control groups allows for researchers to conduct a GIE over a much larger area, lending insights and greater external validity.
GIEs are not without their limitations. Retrospective geospatial data is not available for all sectors, limiting those for which GIE can be used. In addition, some types of development projects—such as those focusing on budget support or governance—do not have definite physical locations, and so cannot be analyzed through a GIE.
GIEs on the ground
The best way to understand GIEs is to see one in action.
Take the Demarcation of Indian Territories Project (PPTAL) in the Brazilian Amazon, co-funded by the World Bank and KfW to formalize and demarcate indigenous communities’ land rights over nearly 40 million hectares of forest. BenYishay and a team of researchers from AidData and KfW conducted a GIE seven years after PPTAL closed, with the goal of determining the project’s impact on deforestation. While PPTAL was not designed with an impact evaluation in mind, the team was able to gather project data from PPTAL’s implementing and funding agencies and join it with high-resolution satellite imagery to conduct a GIE evaluating the changes to forest cover over more than three decades. That initial GIE, originally published as an AidData Working Paper, found that PPTAL produced little evidence of conservation effects.
However, a follow-up GIE on PPTAL’s effects on land-related conflicts, recently published in the Journal of Economics and Environmental Management, reveals that the incidence of land-related conflict significantly fell in PPTAL communities, and that these effects grew over time. The PPTAL case shows how not just one, but multiple GIEs can be conducted for development projects, even if the project was not designed to incorporate such an evaluation in the first place.
GIEs are powerful methods of project evaluation that can match the strength of RCTs while addressing some of their shortfalls. While GIEs are not suitable for all development projects, they allow researchers to geospatially analyze project impact in a quick and cost-effective way. For more on GIE methods, their advantages and disadvantages, and their use cases across countries, sectors, intervention types, and development organizations, dive into the full primer on GIEs.
