Cliquez ici pour lire ce blog en français.
Even before COVID-19 imposed severe travel constraints and imploded budgets, collecting household survey data to evaluate development projects was costly and challenging. In partnership with Development Gateway, AidData has been implementing a USAID-funded project to bolster open health data in Côte d'Ivoire. One part of this project has involved AidData exploring techniques that will enable program evaluators to analyze the impacts of projects, even as access to field data tightens.
These techniques leverage alternative sources of data called predicted surfaces. One example of a predicted surface is data from the Institute for Health Metrics and Evaluation (IHME) that essentially joined together—or interpolated—a very large number of survey microdata and remotely sensed data sources to then predict the number of adults aged 15-49 who are living with HIV. These estimates are mapped to 5km grid cells across 47 countries in Africa—so for each 5x5km area of land, IHME has an estimate of the prevalence of HIV among adults there.
A new white paper from AidData explores the promises and limits of predicted data sources like these for development evaluation. Reducing the Costs and Barriers to Evaluations Using Geospatial Data: New Methods with an Application for HIV/AIDS in Côte d'Ivoire aims to help evaluators, funders, and project implementers understand their options for combining multiple rounds of survey data and geospatial data to evaluate projects, particularly those in the health sector.
Although predicted surfaces can’t answer questions at the household level—they lack the precision of data collected house by house in the field—they are a promising way to analyze a development outcome at coarser levels (such as the HIV rate of a single village, or the vaccination rate of everyone within 5km of a town).
"The high cost of collecting household survey data means that these demographic and health surveys are conducted only once every three to five years," notes Kathy Nolan, AidData Senior Research Analyst and a co-author of the white paper. "Impact evaluations place additional data burdens on development projects. To perform a valid and rich analysis of a development intervention, we often need to establish a counterfactual—what would have happened in absence of the intervention?—and for that, we need the project to have collected data at periodic intervals: before, during, at the end, and after. But this isn’t always logistically or financially feasible.”
Lack of data at these milestones can stymie an impact evaluation, but predicted surfaces can help. "Using newly-created predicted surfaces can help us overcome this “missing baseline” scenario, where no baseline survey was conducted in both treatment and comparison locations," adds Ariel BenYishay, AidData’s Chief Economist and Director of Research & Evaluation, and a co-author of the whitepaper.
In the white paper, AidData reports the results of simulations using predicted surfaces for three scenarios commonly faced by program evaluators, using the challenge of estimating HIV and vaccination rates in Côte d'Ivoire as a case study.
- Missing follow-up: Survey data is available for the baseline, but not a follow-up. Instead, predicted scenarios are used for the follow-up.
- Missing baseline: Survey data is available for the follow-up, but predicted surfaces are used to substitute for a missing baseline.
- Missing both: Predicted surfaces are used for baseline and follow-up milestones.
The advantage of using household survey data (instead of building a predicted surface) is granularity. Demographic and health surveys collect data on many attributes, so multiple attributes such as age or income can be reliably linked to a single data observation. But a big disadvantage is coverage—household surveys are often only available for very specific timeframes and locations.
For predicted surfaces, the advantages and disadvantages are flipped. Coverage is an advantage: "as an example, researchers at IHME developed predicted layers for a comprehensive set of development indicators for all of sub-saharan Africa," explains BenYishay. But the lack of granularity is a disadvantage for predicted surfaces. "Predicted surfaces can only capture change at a certain level," BenYishay notes. "For example, you could only use a predicted surface, like the IHME indicators, to explain differences between two 5km areas, but not within a single 5km area."
In the specific problem context of predicting HIV and vaccination rates in Côte d'Ivoire, the "missing baseline" scenario had the least statistical uncertainty, effectively melding the advantages of using both survey data and predicted surfaces. "Importantly, with this method, evaluators could potentially use a predicated surface for their baseline estimates and use their own survey for an endline,” explains Nolan, "which would cut the cost of surveying in half."
Notably, the scenario in which only predicted surfaces were used actually outperformed the scenario where survey data was available for the baseline. This may be because, when estimating HIV indicators, having highly-specific survey data from a survey at the beginning of a project (but not at the end) effectively gives you the drawbacks of using both types of data.
Tradeoffs—between the more frequent and wider geographic coverage of predicted surfaces and the unique ability to detect statistically meaningful differences when granular household survey data is available—are one key theme of the AidData researchers’ white paper. Another are twin challenges of statistical techniques needed for the simulation and data integration.
Statistical analysis rests on the critical assumption that a given sample of data can represent a certain population with a predictable level of accuracy. For demographic and household surveys, this implies researchers have a minimum viable number of observations to be able to control for unrelated factors which might account for trends reflected in the data.
When data is limited, researchers may perform simulations that "randomly sample from the sample" to generate enough observations to make a statistical prediction. Although these simulations result in a higher range of uncertainty than an estimate from a large number of observations, simulation techniques do enable trends (or lack thereof) to be visualized and analyzed. The level of uncertainty can also be managed: for example, predicting the HIV rate for a certain demographic within an entire district will vary less than a similar prediction for each and every sub-district.
The statistical simulations profiled in the white paper are analogous to how a predicted surface is created from geospatial data using an interpolation method. Both statistical simulations and geospatial interpolation (1) take a small number of data points, (2) make some assumptions about how these points might be related, and (3) extrapolate the results of an algorithm to a wider or more general level. For example, if the HIV rate in a certain 5x5km area is not available, an interpolation method might impute or substitute in this missing value based on an inference about the HIV rates in neighboring areas where data is available.
The second major challenge to mainstreaming predicted surfaces is data integration. "We had to create indicators that matched across a high number of data sources," explains BenYishay. "This means that a calculation to create a metric like vaccination using one data source, for example, also needs to match the calculation of vaccination rates using a different data source with slightly different attributes."
The white paper explores the opportunities, challenges, and inherent tradeoffs if predicted surfaces are to be mainstreamed in development impact evaluations. BenYishay highlights the opportunity: "Predicted surfaces are able to cover entire regions and countries and include a growing number of important development metrics, such as population estimates, use of health services, birth and pregnancy outcomes, child health inequality, educational attainment, and more."
The statistical and integration challenges are solvable, and, as the availability and precision of predicted surfaces improves, program evaluators will have better tools to pinpoint where a specific intervention is actually working. "The new and innovative research technique presented in this paper can reduce the cost of implementing an impact evaluation without reducing the rigor," concludes Nolan.
