Impact evaluations of development projects in fragile states—where collecting survey data on program outcomes is often too dangerous or prohibitively expensive—are particularly challenging. But thanks to the growing availability of remotely-sensed data and satellite imagery, impact evaluations in conflict-affected areas increasingly can be performed even in the total absence of survey data. A case in point is a new study by AidData using satellite measures of vegetative greenness (NDVI) to evaluate how a large-scale USAID infrastructure investment to improve irrigation canals in Afghanistan affected the productivity of nearby farms.
AidData’s evaluation finds that improvements to irrigation canals increased the greenness of surrounding fields by 15% on average over the study period. While there’s additional uncertainty in correlating increases in vegetative greenness to wheat yields in Afghanistan, one estimate drawn from a 2008 national-level study suggests that these changes in greenness correspond to substantial improvements (perhaps as high as 48%) in actual wheat yields in the fields surrounding the OFWMP-supported canals.
Wheat is a staple crop in Afghanistan, yet the country currently must import a third of its wheat just to meet domestic consumption. Improving the yields of staple crops like wheat is therefore critically important to the country’s food security and has long been a priority for international aid agencies as they seek to provide farmers with profitable alternatives to opium.
AidData collaborated with USAID/Afghanistan’s Office of Agriculture to evaluate one such program, the On-Farm Water Management Project (OFWMP), that aims to improve agricultural productivity through more efficient use of water—key in an arid climate like Afghanistan’s. USAID and the World Bank co-financed the first $25-million phase of the project, which provided water to 19,000 hectares of land through the rehabilitation of 80 irrigation canals in a third of the country’s provinces between 2013 and 2016.
As the US and other donor governments have sought to bolster Afghanistan’s agricultural sector through projects like the OFWMP, prior studies have estimated that wheat production nationwide grew by 7.3% each year from 2002 to 2017. Yet these earlier studies could not determine the net, attributable impact that specific infrastructure improvements had on agricultural productivity. AidData's evaluation not only finds that the OFWMP had positive effects on crop greenness overall, it also indicates that impacts accrued disproportionately to those areas closest to the treated canals—in particular, the starting points of canals. Fields closest to the start of an OFWMP-improved canal showed approximately a 67% greater increase in greenness than fields farthest from a canal start.
To determine impact, a geospatial impact evaluation like the OFWMP study requires two types of location-based data: program treatments (in this case, where and when improvements to irrigation canals were completed) and outcome observations (how much greener nearby fields became). Here, AidData measured crop productivity using NDVI, a scale that provides a standardized measure of vegetative greenness. Courtesy of the Landsat satellite, we have granular observations of NDVI that go back in time long before canal improvements began. These NDVI observations are incredibly precise, enabling the observation of changes in greenness in the cultivated areas around canals at the 30 meter x 30 meter level.
Getting program locations required more effort. USAID provided AidData with 80 high-resolution static maps of canals that were rehabilitated through the OFWMP. Over several weeks, a team of students at William & Mary digitized and geo-referenced these canals and their surrounding cultivated areas. These data were then linked with administrative program data to identify the exact timing of each canal improvement.
A geospatial impact evaluation like this leverages an enormous amount of data to create a statistical model that—similar to real-world treatment and control groups in a randomized trial—adjusts for many factors. The OFWMP evaluation feeds over nine million observations of NDVI over an 11-year period into a model that evaluates whether a measured change in greenness is due to the program or another factor that could affect vegetative growth, such as weather, broader environmental changes, or even a field’s initial level of productivity.
This evaluation is part of a growing body of work that is dramatically expanding the available tools for impact evaluations in contexts where evaluators face barriers to collecting survey data—whether that limitation is due to conflict in-country, high costs of collecting data on-the-ground, or the fact that a program is already underway or completed and no controlled trial has been planned or data specifically collected for a control group. Remote-sensing technologies will increasingly enable impact evaluations of development projects in areas where survey enumerators can't feasibly go.
