In its World Development Report 2018, the World Bank put a global spotlight on a chronic “learning crisis” in many countries: children may be in school, but they are learning little. As they attempt to improve learning outcomes, education decision-makers must also overcome a crucial measurement gap--only half of all low- and middle-income countries have adequate metrics to monitor learning achievement at the end of primary and lower secondary school.
Yet, even as this lack of data is a problem across developing countries, we have limited visibility into whether and how decision-makers use existing education data in their work. Today, AidData and the Brookings Institution launched a joint report, Toward data-driven education systems: Insights into using information to measure results and manage change, to answer this question, using results from two unique global surveys: the 2017 Listening to Leaders Survey and the 2017 Education Snap Poll.
Drawing from more than 300 responses from education decision-makers (including government officials, in-country development partners, and domestic civil society leaders), the report shares insights into the barriers to evidence use and the types of information that these leaders want to improve learning outcomes in their countries.
What data do education decision-makers value and need?
A typical assumption accompanying the release and dissemination of data and analysis is that leaders will rely on this information as they assess anticipated benefits and costs of a decision.
But do they? We find that having enough information is rarely the decisive factor in making most education decisions; rather it is sufficient government capacity to implement policy and programmatic changes, that matters the most. This is not to say that information is not important, merely, that it sits in the supporting rather than the lead role.
So, even as efforts are in progress to build government capacity, what can funders and producers of education data do differently to ensure information is making its way into decisions? In this blog, we share three key findings from the report.
Program evaluation data is used most often, considered most essential, and decision-makers want more of it.
Information may not be the decisive factor in education decisions, but it is still a variable decision-makers consider: over 60 percent of respondents to our 2017 Listening to Leaders Survey say they use program or project evaluation data in their work. And there appears to be a premium on impact evaluations, as 75 percent of education sector leaders report using this type of analysis. The 2017 Education Snap Poll further validates the importance of this type of data: more than half deem project and program evaluation data to be essential to their work, and a third say they wish more of it existed.
To close the gap between high demand and low supply, funders and producers of program evaluation data should do two things. First, they should increase the geographical and topical coverage of impact evaluations in the education sector, which are presently heavily concentrated in only a few countries and topics.
Second, policymakers’ ability to interpret evidence and link it to policy decisions may also be limited. Producers of program evaluation data or analysis should therefore invest in communicating the results of their research in ways that can be easily understood, while highlighting the generalizability of findings, any caveats that decision-makers should know, and how the findings from the research can be useful in programmatic and policy decisions. Whenever possible, researchers should involve policymakers in the design and execution of evaluations to increase their salience and relevance.
Data needs vary by decision-type; producers should target outreach of their data and analysis to match demand.
Respondents to the 2017 Education Snap Poll could select involvement in ten common education decisions, which we further consolidated into four decision-making domains adapted from the OECD’s Education at a Glance 2012 Indicators: (1) planning and structures, (2) organization of instruction, (3) resource management and (4) personnel management.
Juxtaposing leaders’ responses about what they wished for against the data they deemed most essential for their work, we find that education administrative data, while essential, is more or less a “met need.” Data types that were both deemed as essential to leaders’ work and ranked high on their wish lists represent “unmet needs” and therefore attractive investment opportunities for data producers to increase their impact in response to user demand (see table below).
Those making or shaping decisions related to instructional matters would like more student-level assessment data, and those in personnel management desire more teacher performance data. Unsurprisingly, decisions involving resource management have unmet needs for government budget and expenditure data.
While investment in broad-based dissemination strategies works well for overall visibility, data producers should target data where it can spur action, i.e., each data type should be disseminated in line with the functional roles of decision-makers.
End users care most about improved timeliness and accessibility of data, and prioritize strengthening information management systems.
Asked to identify improvements to make information from education data sources more helpful, decision-makers and influencers expressed a desire for more timely and more accessible data (see figure below). Over half of leaders surveyed also identified efforts to improve data disaggregation, accuracy, and trustworthiness as desirable.
What potential solutions can help realize this fairly long list of desired improvements? Of the seven solutions proposed in the survey, the recommendation to strengthen their countries’ education management information system (EMIS) within the Ministry of Education resonated with the highest number of respondents.
To realize this solution in practice, domestic leaders and their development partners will need to address several common EMIS shortcomings, namely: fragmentation of data collection activities across ministries; unclear protocols for open sharing and dissemination of data; lack of funding; poor coordination among international funders; and a nascent or deficient statistical culture. Ensuring that data are open and trustworthy implies some level of administrative independence from political interests throughout the information cycle. Investments in review and verification processes by a third party will pay off.
Our findings reveal the data needs and preferences of education decision-makers in low- and middle-income countries. More importantly, they present opportunities for data champions and advocates to improve the status quo in terms of elevating the role of information in decision-making. Making future data investments and designing and disseminating data products in line with policy-maker needs can hopefully bring us a step closer to mitigating the “learning crisis” in education.
Read the full report, or check out Brookings' Education Plus Development blog on the 6 key insights into the data and information education leaders want most.
