Only five countries in the world are poorer than Malawi, a landlocked nation in southern Africa with high levels of food insecurity. Despite significant economic and structural reforms in recent years, more than 80 percent of Malawians are still dependent on agriculture for their livelihoods, making the country extremely vulnerable to external shocks. Against this backdrop, USAID’s Office of Food for Peace, in cooperation with USAID/Malawi, began a program in 2009 to reduce chronic undernutrition and improve food security in more than 200,000 households in the most food-insecure areas of southern Malawi.
Read: Long-term Impact Evaluation of the Malawi Wellness and Agriculture for Life Advancement Program
Called WALA, for “Wellness and Agriculture for Life Advancement,” the program entailed an extensive array of activities—on village savings and loans, child and maternal health and nutrition, agriculture, natural resource management, irrigation, and disaster risk reduction—all focused on building households’ capacity and increasing their resilience to external shocks like drought, floods, pests and crop failures, and market failures.
By the end of the program in 2014, it appeared that the WALA target communities experienced a major reduction in child stunting and underweight rates, as well as in the number of communities that reported needing food aid in crises. But the significance and durability of these outcomes was an open question—Malawi had experienced unusually severe climate and pest-related shocks in the years since WALA’s end. More importantly, the performance metrics that indicated an improvement in key outcomes had only been taken in the villages where WALA operated. USAID’s Office of Food for Peace was interested to learn whether the nutritional and other gains in the WALA target communities continued or sustained as a result of the enhanced capacities and positive behaviors.
To answer this question, AidData partnered with USAID’s Office of Food for Peace, USAID/Malawi, Mathematica, the University of Notre Dame Pulte Institute for Global Development, and the U.S. Global Development Lab to carry out a long-run impact evaluation of the WALA program. Without a way to identify comparison groups against which to benchmark the results in treated (WALA target) villages, it would have been impossible to understand whether WALA target villages are doing any better in terms of nutritional and food security outcomes compared to villages that did not receive WALA support.
Our evaluation matched 100 treated WALA villages to 100 untreated comparison villages that had similar stunting rates at baseline in 2009. To identify comparison villages with similar initial conditions to WALA villages, we used 2010 Demographic Health Survey (DHS) data and other remotely sensed spatial data to create a continuous layer of data on child stunting using a specific tool (EBK Regression Prediction in ArcGIS Pro). By checking the DHS data against our survey results, our research team could confirm that the treatment and comparison groups were indeed comparable across a variety of demographic indicators and perform a rigorous impact evaluation of how households fared in the years after WALA ended.
“WALA was implemented in eight of the most food-insecure districts in Southern Malawi, in an effort to alleviate that food insecurity. But what we found was that resilience in the face of repeated, extreme shocks was particularly hard to achieve—WALA households were not able to overcome the severe shocks that occurred after the program ended,” said Kathy Nolan, AidData Senior Research Analyst and WALA evaluation team member. “Going forward, development program implementers will need to pay special attention to these impending environmental disruptions and future shocks that households will encounter as the climate changes. Hopefully, developing and implementing strategies to improve post-project sustainability can help mitigate these shocks for treatment households after a project has ended,” said Nolan.
In their report, the evaluation team found that rates of both stunting and underweight status were not statistically different between treated and comparison villages. The data shows that both WALA and comparison villages were hit equally hard by back-to-back climatic shocks—including severe droughts and floods—and any improvements in resilience from WALA at the project's close were unable to help communities overcome the severe environmental disruptions seen between the end of the project and today.
“Long-term impact evaluations like this really give you the insight to answer: were the outcomes produced by a project sustainable? The hope for all development projects is to create positive, long-term change for communities. The data doesn’t always back that. This evaluation let us see what was sustainable and was not sustainable about the WALA program. And this evidence, in turn, will help inform the Office of Food for Peace and USAID’s future programming of projects,” said Nolan.
Watch: Kathy Nolan, AidData Senior Research Analyst, shared insights from AidData’s work at a Mathematica/USAID forum highlighting the power of long-term impact evaluation.
On the ground and in the field
In order to ascertain WALA’s impacts, AidData and Mathematica Policy Research teamed up to collect extensive quantitative and qualitative follow-up data from both WALA villages and comparison villages.
To get a deeper view of how households were doing in the years after WALA ended, AidData administered quantitative surveys to 21 randomly selected households in each of the 100 treatment and 100 control villages. The survey covered aspects of health, nutrition, agriculture, income, and development project participation, and also included questions on under-five child nutrition and maternal health. As part of the post-program data collection process, AidData also sent enumerators to villages to take anthropometric measurements (height and weight) of Malawian children. This allowed the evaluators to see whether improvements in resilience due to WALA allowed communities to maintain child nutrition gains even in the face of environmental disruption. Using this quantitative data, the research team was able to confirm that comparison villages were not targeted at higher rates by other, non-WALA development projects, as this could have skewed the results.
Kristen Velyvis, a senior researcher at Mathematica, led the effort to collect qualitative data. Surveys were performed in 14 villages, selected on the basis of particularly successful WALA implementation and the largest drops in stunting rates. These interviews covered a wide range of topics—including perspectives on the sustainability of WALA activities and of WALA activities in general, household resilience, coping mechanisms, livelihoods in the face of shocks, diversification of income sources, and reliance on direct food aid.
Collecting data and pinning down geographic locations was not always a smooth process. “Think about the data you need to successfully evaluate the impact of a development program. You probably think of having GPS data on precisely where project locations and surveys took place. In this case, we actually did not have exact GPS coordinates for the locations of villages where WALA was implemented,” said Nolan.
“Instead, we knew the Traditional Authority in which a village was located. That gave us a small enough geographic area to identify WALA villages with certain stunting rates, and then match them to comparisons. So, by all means, collect GPS data on your programs! But the takeaway is that, even without it, there are other ways we can find and match project areas to do this kind of rigorous impact evaluation, even years after the program ended.”
