Source: Billboard photo by Aaron Ross/The Economist; FY13 Côte d’Ivoire Scorecard from MCC
Something odd happened in Côte d’Ivoire during the summer of 2013. Billboards advertising the negative effects of corruption mysteriously appeared throughout Abijdan. As a public awareness campaign, this had mixed results. “It killed my wife” and other oblique references to the illicit behavior were catchy, but many citizens confused the billboards with an ad campaign for the mobile telecom company Orange.
What motivated the Ivorian government to pursue this public awareness campaign in the first place? After some digging, a journalist from The Economist discovered that the Alassane Outtarra administration was seeking to overcome the Millennium Challenge Corporation’s (MCC) anti-corruption hurdle – a bar that low-income and lower-middle income countries must pass to unlock hundreds of millions of dollars in grant assistance.
The Côte d’Ivoire case illuminates a broader trend: donors, international organizations, and NGOs have developed a wide range of policy instruments to change the ideas, interests, and incentives of low- and middle-income countries. In a newly released report, The Marketplace of Ideas for Policy Change, we asked nearly 6,750 development policymakers and practitioners from 126 developing countries which external sources of policy analysis and advice they listen to and why.
Do leaders pay more attention to external assessments that benchmark their performance against other countries, or to assessments that are specifically focused on their own country? The answer to this question is nuanced: country-specific assessment are generally more influential, but it also very much depends on the context.
Country-specific assessments tend to encourage leaders to focus on “nuts and bolts” reforms within their governments – for example, modernizing data collection and information systems or upgrading the skills and competencies of government staff. They also tend to contain specific and actionable policy guidance. A country-specific assessment might, say, provide the Government of Côte d’Ivoire with practical recommendations regarding how it can more effectively enforce anti-corruption legislation.
By contrast, cross-country benchmarking exercises, like the MCC scorecards, rarely provide specific and actionable policy guidance. They also tend to steer governments towards efforts to change the behaviors and incentives of individuals and firms (see Figure 1). The nature of the Ivorian Government’s response to the MCC’s cross-country measure of anti-corruption performance is a case in point. Lacking specific policy advice from the MCC about how corruption should be tackled, the Ouattara administration resorted to a broad rather than a targeted policy intervention (i.e., a public awareness campaign).
The difference in reform focus (highlighted in Figure 1) can be interpreted in one of two ways. A cynic would argue that government leaders do not wish to be held accountable for reform outcomes, so they either use high-profile cross-country benchmarking assessments to create the appearance of reform, or they use country-specific assessments to solve the problems they already want to fix. A more sanguine interpretation is that governments in low- and middle-income countries are often pressured to “run before they can walk,” when they should instead be encouraged to focus on simple, solvable problems before undertaking more complex reforms.
We consider each interpretation in turn.
Do country-specific assessments tend to focus reforms on problems that are easier to solve? Data from the 2014 Reform Efforts Survey (see Figure 2) provide little support for this hypothesis. In contrast to comparative assessments conducted across multiple countries, the problems most influenced by country-specific assessments are not associated with systematically more or less success in reform implementation.
In fact, assessment influence in general does not seem to be associated with the solubility of the problems that are addressed in these assessments. The top four “problem types” most influenced by all assessments – (1) behaviors or incentives of private individuals or firms need to change, (2) insufficient human capital, (3) inadequate data collection or information systems, and (4) inadequate or inconsistent implementation or enforcement – are not always the easiest problems to solve through reform.
So patterns in influence by assessment type do not seem to be easily explained by the ease of reform implementation. Do external signaling dynamics and alignment with domestic policy priorities serve as better explanations?
We asked participants in the 2014 Reform Efforts Survey to identify the factors that best explain the influence of individual assessments. Analysis of the survey data (see Figure 3) suggests that country-specific assessments have relatively more influence when (1) performance on the assessment is directly tied to a financial incentive, (2) the “policy space” needed to find locally-appropriate interventions is preserved, and (3) the government’s sovereign authority to take final policy decisions is respected.
Country-specific assessments are likely seen as more respectful of national sovereignty and policy autonomy because they engage the governments that are being assessed as partners in the measurement and evaluation process. Nearly 95% of the country-specific assessments in our sample draw upon data from the government being assessed, while only 35% of cross-country benchmarking exercises do the same.
We also find support (see Figure 3) for the hypothesis that when cross-country benchmarking assessments, such as the Millennium Development Goals and the MCC scorecards, are influential, it is usually because they serve as mechanisms for low- and middle-income governments to signal their credibility to donors and investors. This evidence is broadly consistent with the more cynical view that developing country officials may use cross-country assessments to self-promote and obtain external rewards without substantially changing their policy priorities or actions.
However, all hope is not lost. While government officials may seek to obtain maximum rewards with minimum effort, we do not find that external incentives and pressures to “game the system” are significantly distorting the domestic policy priorities or reform activities of developing country leaders. Figure 3 demonstrates that cross-country benchmarking exercises are influential, first and foremost, because they help governments to more fully appreciate policy problems and potential solutions to these problems. The weight of the evidence also suggests that external incentives are probably secondary considerations, after government decision-makers first determine whether an assessment will support the domestic problem-solving process.
Additionally, cross-country benchmarking assessments appear to have some positive and underappreciated effects (see Figure 4) on the reform process – namely, helping civil society organizations and journalists advocate for reform and shoring up the positions of reformers within governments by allowing them to build broader constituencies and coalitions in support of reform.
These findings from the 2014 Reform Efforts Survey should give pause to those who are negotiating the soon-to-be-established Sustainable Development Goals. While a great deal of effort is currently being devoted to the identification of technically appropriate and politically feasible indicators and targets, UN negotiators would be wise to think long and hard about how the modalities of performance assessment that they choose may influence the incentives and behaviors of the governments they are monitoring and evaluating.
The international community would serve itself well by listening carefully to the makers and shapers of policy in low- and middle-income countries. A reasonable place to begin would be with the 6,750 development policymakers and practitioners who participated in the 2014 Reform Efforts Survey. We have summarized their experiences and insights in this new report, and through a series of follow-on surveys and snap polls will continue giving voice to this uniquely important group of leaders in the developing world.
