Studying satellite imagery to evaluate development projects for climate resilience and other impacts is rare. AidData, with the German Institute for Development Evaluation (DEval), spent some two years pushing the envelope in this direction, as well as incorporating a multitude of diverse datasets as they looked at the impacts in nearly 1,000 distinct locations in rural Mali where small-scale irrigation was introduced over the past two decades.
“Past evaluations of irrigation interventions in the agricultural sector have rarely looked at small-scale irrigation (think more single pumps leading to fields than huge dams or large concrete canals), and certainly not over any kind of long-term timeline,” says Ariel BenYishay, AidData Chief Economist and Director of the Research and Evaluation Unit. “But smaller-scale irrigation is much more feasible in many areas and may be more adaptable to climate change. Understanding how such irrigation could lead to food security and health improvements in the face of climate change remains crucial. That’s what we attempted to achieve here.”
Results from the geospatial impact evaluation (GIE) are now out; the discussion paper, “Does Irrigation Strengthen Climate Resilience? A Geospatial Impact Evaluation of Interventions in Mali,” is now available online.
- The projects, funded by German development agencies, reduced the vulnerability of farming communities to some of the extremes of climate change.
- The introduction of irrigation led to substantial increases in agricultural production on nearby fields, with these gains persisting even a decade later.
- Crop diversity increased substantially after the interventions.
- As a result of increased food production, children in nearby communities are less likely to be stunted or severely underweight.
- Conflict risks also decreased in communities near irrigation projects.
This GIE and discussion paper also contributed to DEval's broader Evaluation of Interventions for Climate Change Adaptation.
The climate issue is critical in Mali, exposed as it is to dry, hot conditions and extreme events like droughts plus heavy and irregular rains. Some 80 percent of the population is engaged in agriculture. The findings from Mali could be applied to other parts of the Sahel, including parts of Niger, Nigeria, Burkina Faso, Senegal, and Chad.
“Rigorous impact evaluations of small-scale agriculture have mainly explored the short-term effects of these projects,” says BenYishay. “Only a few have explored more long-lasting effects, and this is critical in assessing whether vulnerabilities have been permanently reduced and whether farming communities have become resilient in dealing with future climate shocks and stressors. Short-term effects that seem to make a positive contribution might turn out to be inadequate strategies for adapting to future climate conditions and may even lead to maladaptation.”
The researchers found that increased accessibility to water and resilience to rainfall shocks translates into increased agricultural productivity. These gains begin in the season following the completion of the irrigation infrastructure and continue over the ensuing 10 or more years. There was more water available prior to the rainy season, and more vegetation greenness visible both prior to and during the rainy season.
“The staggered roll-out of irrigation projects, plus repeated observations over 20 years, allows us to compare the pre- and post-irrigation outcomes of locations while adjusting for confounding factors,” said AidData Research Scientist Rachel Sayers. “We found sustainable effects, which are an important precondition for resilience. However, the ultimate test for the extent to which irrigation contributes to beneficiary communities’ climate resilience will be the droughts and erratic rainfalls, which are predicted to increase in intensity and frequency in the future.”
The team drew on geocoded project locations, survey data from the Demographic and Health Surveys and the Living Standards Measurement Study, data on child nutrition and health, the Ethnographic Atlas, as well as other sources. The researchers used satellite imagery from before and after project interventions. These included measures of agricultural production using Google Earth Engine (GEE) and Landsat satellite imagery, combining bands from the visible and different parts of the infrared portion of the electromagnetic spectrum to estimate, for example, plant water content. Shapefiles of project clusters were imported to GEE to develop water and vegetation indices from 1986 to 2021.
The team used very high-resolution (VHR) and high-resolution (HR) satellite imagery to assess ecological impacts. The spatial resolution of VHR images range from 0.3 meters to 1 meter. Before carrying out the visual interpretation of remotely sensed satellite imagery within each project site cluster, the team overlaid the shapefile (i.e., the geometric shape outlining the project area) on Google Earth Pro to digitize existing farm boundaries within each project using the historical VHR imagery archive. Digitized farms helped assess the approximate percentage of the land that was cultivated and subsequent ecological impacts within each project site cluster. To visually analyze both HR and VHR imagery, downloaded images and digitized farm boundaries were brought into QGIS, an open-source geographic information system software.
One issue the researchers looked at was whether irrigation resulted in more, or less, soil erosion. They were only able to do this because not all soil looks the same from space. Inherent differences in color and pattern between topsoil and subsoil allow experts to use VHR data to assess whether erosion is taking place and detect rills (small streams), gullies, and stream channels, for example.
“The spectral reflectance of soil types in the visual part of the electromagnetic spectrum varies substantially,” says Kunwar Singh, AidData Senior Geospatial Scientist. “If the topsoil is severely eroded due to excess irrigation, a significant portion of organic matter is lost, which exposes the lighter color of the subsoil layer. Subsoils have considerably higher spectral reflectance than the darker color topsoil. These variations create color contrast in remotely sensed imagery.”
Results were somewhat inconclusive—there was some evidence that irrigation may have reduced soil erosion by water and improved soil moisture, but visual interpretation of VHR images also suggested that soil erosion due to excessive irrigation frequently occurred around project clusters. In addition, seasonal gully and stream-channel soil erosion may have led to algal blooms at Niger riverbanks. However, whether these erosions are due to excess irrigation or the flow of rainwater from farms to the river is not clearly understood.
The findings come as AidData also launches an online course on GIEs for agriculture projects. In addition to video tutorials, the toolkit includes slides, sample code, related articles, and guidance documents that cover a range of topics needed for these evaluations, including geospatial data and analysis, as well as causal inference and impact evaluation approaches.
“Our study with DEval adds to the young but growing field of GIEs, by focusing on longer time frames than most GIEs and by using a multitude of sources of remote sensing data,” said BenYishay. “We hope that by having more use cases like ours, sharing the results, and putting tutorials online about how to implement GIEs in the agriculture space, we will also see more organizations use the methodology themselves.”
The AidData-DEval GIE looked at three types of irrigation—small-scale pumps, large-scale gravitation systems, and adapting floodplains to strengthen climate resilience. It’s a constant battle: rising temperatures and decreasing rainfall mean ever more water is needed for agriculture across the Sahel.
“This is also a rare study on the impacts of irrigation interventions in the midst of armed conflict,” said Mascha Rauschenbach, evaluator/team leader at DEval. “We know one strength of GIEs is that they are a useful tool for evaluations in conflict-affected regions, where on-the-ground work may be impossible.” For the Mali study, the team integrated geocoded conflict data from the Armed Conflict Location & Event Data Project (ACLED) into statistical models to assess if projects are affected by ongoing conflict and vice versa—plus whether the projects themselves may impact conflict risk and intensity, for good or ill. AidData has done one similar study previously, looking at the rehabilitation of irrigation infrastructure in Afghanistan.
“The impacts of climate change, particularly in conflict contexts, are putting increasing pressure on development cooperation and the international agenda on aid effectiveness,” said Sven Harten, Head of the Competence Centre for Evaluation Methodology and Deputy Director of DEval. “This gap between increasing relevance and low evidence needs to be further reduced. At the national and international levels, therefore, the number of rigorous evaluations dedicated to the impacts of climate adaptation interventions in the context of fragile states using geospatial data should be increased.”
”Trying to evaluate interventions on the ground in regions impacted by conflict is potentially dangerous, but there is also risk involved in assuming that what works in a non-conflict environment is going to transfer or apply to a conflict zone,” said Seth Goodman, AidData Research Scientist. “Conflict may limit the effectiveness of intervention measures by directly interrupting project activities. Conflict may also result in the destruction of harvests, aggravating climate-induced vulnerabilities. Lastly, the intervention might also exacerbate the existing conflict by, for example, creating disputes over the allocation of land to be irrigated. In this case, we were able to assess impacts with reasonable rigor in a context where data collection on the ground would not be feasible due to security concerns.” DEval was, however, able to conduct focus groups in some parts of Mali to better understand the research findings and ‘ground truth’ the evidence uncovered.
“This GIE shows that even in conflict contexts, where on-site data collection is not possible due to the limited security situation, development impacts (such as with regard to strengthening climate resilience) and even potential unintended effects (such as the exacerbation of conflicts by the interventions) can be reliably assessed,” said Martin Noltze, Senior Evaluator at DEval.
With so many outcomes studied, it’s perhaps inevitable that almost as many questions or new avenues for research were generated as answers. For example, the team used both reported survey measures and biometric measures of child nutrition and health from the DHS to consider the effect of the projects on food security and food composition in project areas. As noted above, while both child stunting and child wasting decreased because of treatment in areas close (0–4 km) to project locations, further away (4-6 km) there was actually an increase in child stunting and wasting. Similarly, irrigation did not result in higher employment rates or other household materials gains. And while conflict decreased near irrigation sites, it appeared more likely to occur in the peripheral surroundings (4-6 km) of completed irrigation sites, suggesting that conflict may have shifted rather than declined overall.
Also disappointing, the study did not find consistent improvements in women’s overall decision-making power. But there were changes in views on intimate partner violence (IPV) for women who were 2-4 km away from the irrigation sites. The introduction of irrigation appears to lead women to view IPV as more problematic and less frequently justified, suggesting some limited improvement in women’s empowerment and the potential to positively impact gender equality.
The issue of women in agriculture—and the connections to climate, economic empowerment, and food security—has engaged AidData researchers in the Gender Equity in Development program for the past few years across several projects. “Mali is a good case for why this kind of research is needed,” said Sayers. “Women are increasingly responsible for their family’s food security, yet many rural communities remain male-dominated hierarchies, despite the fact that many men have migrated for work, are engaged in conflict, or have passed away because of it. At the same time, women farmers in Mali are penalized, with less access to capital to take out loans and repay credit; they hold less land than men; and the land they do own is of poorer quality and with a less secure tenure than that owned by men.”
In April, USAID Administrator Samantha Power announced the launch of Generating Resilience and Opportunities for Women (GROW), a new agency commitment to “tackle urgent challenges women are facing in food and water systems, including climate change, while unlocking opportunities for women to advance economically – ultimately benefiting their families, communities, and societies at large through improved food security, resilience, and economic growth.”
In addition to BenYishay, Sayers, Goodman and Singh, the AidData team implementing the GIE included Junior Data Analysts Madeleine Walker and Christian Baehr.
DEval, founded in 2012, is an agency rare in development circles—a stand-alone organization responsible for the independent analysis and assessment of German development cooperation interventions, mandated by the German Federal Government. This includes evaluations of Germany’s important climate change adaptation portfolio. Results of the research are being shared with the members of the reference group of DEval’s evaluation and their development partners, including the Federal Ministry for Economic Cooperation and Development (BMZ) and the International Climate Initiative (IKI) under the leadership of the Federal Ministry for Economic Affairs and Climate Action, in close cooperation with its founder, the Federal Ministry for the Environment; Nature Conservation, Nuclear Safety and Consumer Protection; and the Federal Foreign Office, the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) and KfW Development Bank (KfW).
The authors have submitted a version of the joint paper to academic peer-reviewed journals.
