When conducting surveys that will feed into a program evaluation, who should be interviewed? Survey implementers and researchers often select the “head of household,” under the assumption that this person is likely the most knowledgeable. But what if this is wrong? How do we know that the “head of household,” frequently a male household member, truly has the most accurate information?
The answer to this question is important because household-level surveys are frequently used to understand the impact of projects and interventions on everything from asset ownership, agricultural plot productivity, livestock productivity, time use, and beyond. When surveys are conducted with only one household member, it’s possible that project outcomes will be mismeasured or biased. Understanding the extent and nature of measurement error in surveys and how it relates to gender is key to improving our measures of gender-based outcomes—thereby boosting policymakers’ and practitioners’ ability to understand the full impact of policies for reducing gender inequality.
With funding from Innovations for Poverty Action (IPA) and the William & Flora Hewlett Foundation, we tested this assumption in northern Ghana. We build upon existing research finding measurement error in individual responses to agricultural surveys, such as overestimating the size of a particular plot of land. Research has also shown that husbands and wives often give different responses to questions about an objective measure in agricultural surveys. For example, when asked about the profit from the sale of the household’s maize harvest, different answers may be given.
For our study, we first wanted to understand the size of this measurement error. Once we had quantified its magnitude, we then wanted to know whether it differs on the basis of gender, intra-household dynamics, and bargaining power. This blog, the latest in a series on our work leading Gender Equity in Development Initiative, summarizes the topline findings from our study (see our previous blogs here).
In our latest study, we conducted household surveys in three regions in northern Ghana, totaling nearly 2,500 surveys in over 1,200 households. The surveys collected information on demographics; livelihoods; water access and sanitation; community involvement; trust in government; both current and retrospective information on agriculture and irrigation; time use; and decision-making and empowerment.
Using a random walk pattern, we selected one male and one female partner, both of whom make decisions over household agriculture plots. Couples could be married or not; in the case of polygamous households, we randomly selected one woman. The selected man and woman separately answered the survey in different locations at the same time to prevent them from overhearing each other’s answers. After surveys were completed, the interviewer accompanied the respondent to each of the household agricultural plots within an approximate 30-minute walk of the household to gather GPS information on plot boundaries. A subset of our sample was then selected to play a series of behavioral games to reveal information about intra-household bargaining and decision-making.
We used the GPS information to obtain objective measures of the plot’s size, distance from the household, and other remotely sensed data. By triangulating the household surveys, outcomes from behavioral games, and this GIS-based information, we were able to understand the magnitude of measurement error in self-reports about respondent’s own agricultural plots and in reports about their partner’s agricultural plots.
We expected to see differences between reports from male and female partners and the GIS-based information for three potential reasons. First, household members may specialize on tasks, leading to a lack of information about what a partner is doing on their own plot, as well as general recall bias. Second, because we asked for self-reports over the past three growing seasons, household members may have difficulty recalling precise information. Third, it may be the case that partners are strategically concealing activity from one another, leading to differing answers. We use the GIS-based data to provide an independent measure that allows us to test whether or not gendered differences in reporting are due to household dynamics, including gender-based dynamics, that influence intra-household bargaining power.
But even before we looked at the answers to survey questions about plot characteristics, we were surprised to see disagreement simply on the number, location, and boundaries of plots—for instance, over a quarter of couples disagreed on how many plots their household had. Part of this disagreement stemmed from differences between how men and women walked the plot: sometimes a woman considered a field to be one plot, but her partner considered it to be two plots. Couples also disagreed on plot location and boundaries, making it difficult to easily match surveys responses from each of them to the correct plots.
What’s more, even when couples did largely agree on these basic facts about their plots, we find significant disagreement over ownership roles. Over half of couples disagreed on whether the partner had sole, partial, or no ownership role over a particular plot. These are striking results, given that in order to conduct agricultural surveys, we first need to know how many plots there are, where they are located, and who has ownership rights over them!
Diving deeper, we also find differences between the plot size the couples reported in the survey and the plot size based on the GPS data. We find that men who report playing a role in decision-making report larger plot sizes in the surveys than what we measured during the GPS ground truthing. Yet, we do not find similar results for women; instead, we find that women who say they play at least some decision-making role report plot sizes that are relatively closer to the GPS measurements. To better understand these results, we examined whether gendered power dynamics in the household could explain these differences, but we found no relationship between the dynamics and plot size reports.
Looking at ownership roles, however, we see a different picture. When compared to the GPS plot measurements, women report larger sizes for plots over which they are at least partial owners, though this is not statistically significant. When we add in our measures of intra-household power, we see that women who have higher levels of intra-household power report smaller plot sizes compared to the GPS-measured plot sizes. However, this gap disappears for plots on which women report some ownership status.
Our research in northern Ghana indicates that researchers and evaluators need to seriously consider the methods by which agricultural surveys are conducted. Though it may seem easier to ask the “head of household” to answer surveys, we provide evidence that responses to survey questions differ based on who is asked and that this measurement error is, in part, driven by gendered differences and power dynamics within the household. Surveying only one respondent per household, especially if head-of-household respondents are overwhelmingly male, may unintentionally produce biased data on which programs and policies are being created.
More research is needed to identify low-cost identification methods for selecting respondents across households that will provide the most accurate data possible. AidData continues to seek methods to reduce measurement error both effectively and efficiently within the constraints of project budgets, with a particular focus on incorporating remote sensing and machine learning to complement traditional survey measures. In particular, AidData’s Gender Equity in Development Initiative is dedicating significant resources to uncovering how remote sensing can be utilized to observe and measure within-household variation in poverty, access to markets, agriculture, and other relevant outcomes.
