At our recent GeoField Convening in Rome, we met with colleagues from across the globe at the headquarters of the UN’s Food and Agricultural Organization (FAO) to discuss the opportunities and the challenges for integrating Earth Observation (EO) into impact evaluations, especially for urgently needed agricultural and climate-related programming. While we drew many takeaways from the gathering, we wanted to share a few of the highlights. We also encourage readers to submit manuscripts to two special issues we’re editing of the Journal of Development Economics and the International Journal of Applied Earth Observation and Geoinformation, both related to integrating Earth Observation into impact evaluations.
We share an urgency to learn
The past summer was the hottest three-month period ever recorded in human history. Farmers and herders around the world seek sustenance from a delicate mix of earth, sunshine, and water, and they’re facing increasingly variable and challenging conditions. There is an imminent need for the global community to quickly learn about which interventions and programs can best support agricultural producers in becoming resilient to climate change. Impact evaluations have long been a cornerstone approach to understanding a development program’s effectiveness—yet, they are still not nearly as prevalent as necessary for this learning.
Why are impact evaluations still not conducted frequently enough? Collecting data by visiting and interviewing people remains costly, and it can be particularly difficult in fragile environments or contexts with recent or ongoing conflict. There are also important design requirements for evaluations to be able to determine causality and these often rule out potential opportunities for evaluation and thus slow down learning. Being able to measure climate-related effects is also critical—but this is often difficult for individual survey respondents to perceive, either because these changes are not easily observable or because they occur over longer timescales.
We’re making large leaps forward
In the meantime, there have been major leaps forward in the capabilities and spatiotemporal coverage of Earth Observation (EO), both through the launches of new sensor and satellite constellations and in drawing key measurements from the imagery and data these sensors collect. These new data have a variety of major benefits that can help us address the limitations to impact evaluation approaches. They offer much more frequent observations, often at monthly, weekly, or even daily time scales. This is particularly important for being able to track the “long tail” of impacts from development investments that have already been made.
Again, if we’re trying to learn faster, a natural place to look for these opportunities is among investments that have already been made. What we’re trying to understand, however, is not just the short-term impacts during the two, three, or four years when a project was being carried out, but long after—often a decade or more after—those investments were made. Moreover, the fuller time series offered by EO also allows us to create retrospective evaluation designs that are more rigorous, by drawing comparison groups that we can be more certain looked consistently similar to the treated groups prior to the interventions. For example, Landsat series imagery goes back decades, so it allows us to identify comparison groups whose time paths closely match those of treated groups.
We can see an example of these benefits in our team’s recent study of small-scale irrigation support in northern Mali (see our blog post here). Using a variety of satellite data (i.e., Landsat and Sentinel imagery), we were able to trace the crop productivity and water availability for each of nearly 800 irrigated perimeters for every year and at critical season-specific windows.
This dense time series enabled us to observe the effects of investments even 10-15 years after the irrigation was initiated, as the graph below demonstrates. Moreover, it allowed us to draw comparisons year-by-year (using panel econometric methods) between sites that had already been irrigated and those that would soon be irrigated but had not yet been. The graph below also shows (in the red pre-trend line) that there was little difference between the sites in the years prior to irrigation, giving us confidence that we can causally attribute these changes to the program itself. The situation in northern Mali also made ground-based survey data collection very challenging over the past two decades—including years in which conflict prevented field teams from carrying out standard household surveys in much of the region. Remotely sensed satellite data offered us a window into agricultural conditions even during such difficult-to-survey times.
Impacts of small-scale irrigation on crop greenness in northern Mali
We have widened the scope
EO data can broaden the scope of what we can observe to include key environmental and climate-related conditions, as well as disaggregate better by gender. For example, in our study in northern Mali, we used very high-resolution imagery (e.g., 50cm to 1m spatial resolution) to identify potential erosion alongside river banks that may have occurred when water ran back from the fields down to the river. We can then estimate how frequently this happens, and whether it is linked to the onset of nearby irrigation. (In this particular study, we actually found such erosion was not very frequent).
Learning from impact evaluations using previous approaches is also often limited in terms of gender equity considerations. In many cases, survey sample sizes are designed to detect overall effects across a population, and doing meaningful gender disaggregation—to determine whether programs impacted men and women differently—is beyond the statistical power of many studies. But what if EO could complement ground-based survey data and increase the sample sizes in ways that would let us do better gender disaggregation?
What does it take to truly integrate Earth Observation and impact evaluation?
One of the most essential pieces of the puzzle is program data that contains the specific locations and specific timing of the interventions. Among the very first tasks when we embark on a potential partnership is to jointly explore the geospatial data to ensure it has the requisite details and accuracy. So, if you’re interested in using EO for impact evaluation, it’s often best to begin by looking for (and at) the geospatial data on the program to be evaluated.
There are some great resources already out there on geospatial impact evaluation (GIE) techniques using EO, including:
- Our AidData Toolkit for Agricultural Geospatial Impact Evaluations, which includes videos, readings, and sample code
- Geo4Dev training modules (including ones on crop mapping and classification) provided by CEGA at UC-Berkeley, New Light Technologies, and 3ie
We’ll be looking for more opportunities to broaden our community integrating EO and IE over the coming year. Please don’t hesitate to reach out to us with any questions or ideas!
