Over the last few weeks, we have been exploring whether there are new insights to glean from the World Bank's massive evaluation dataset. The dataset consists of nearly 10,000 World Bank projects with discrete categorical ratings for variables such as 'Quality at Entry', 'Quality of Supervision', 'Bank Performance', 'Borrower Performance', and 'Project Outcome'. One of the advantages of the dataset is that it allows one to explore both the project-level and country-level determinants of project performance.
In this post, we set out to assess the impact of project supervision on final project outcomes as well as the relative influence of country-level factors, such as corruption and government stability. The first potential correlation we examined was between 'Quality of Supervision' (QOS) and final project outcome. Consistent with the approach taken in an earlier post on this blog, we converted the Bank's six-point QoS and project outcome measures into binary (satisfactory/unsatisfactory) variables. Because the large majority of World Bank projects recorded in the database took place between the years 1984 and 2009, we excluded all prior years from our analysis. All borrower countries that did not have QoS and project outcome data for 10 discrete country-years during this timeframe were also excluded, reducing the sample size to 72 countries. Remaining project QoS and outcome values were averaged separately at the country-year level; these country-year averages were then averaged at the country level. Each recipient country was thus assigned a pair of unique QoS and outcome ratings between 0 (unsatisfactory) and 1 (satisfactory), which are displayed on a scatter plot below.
The results were not surprising: better project supervision generally yielded better project performance. But what else might help determine the success of a project? We used data from the PRS Group’s International Country Risk Guide to assess several contextual factors that may compromise project supervision and/or project outcomes: corruption, government stability, democratic accountability, bureaucratic quality and socioeconomic conditions. Specific country-year scores for each indicator were averaged for the period 1984-2009.These five composite indicators were then compared to their respective QoS and project outcome scores.
Several interesting trends emerge for borrowers with at least ten years of project outcome data (92 countries total) and QoS data (72 total). Only the socioeconomic conditions indicator had a significant impact on the project supervision: the better the socioeconomic conditions in a given country, the higher the QoS. Other country-level factors, such as government stability, did not appear to significantly influence QoS scores.
We obtained stronger results in a similar analysis of project outcomes. Whereas the borrower government’s level of stability still has no apparent role in determining a project’s success or failure, all other factors seem to have an impact; better socioeconomic conditions, lower levels of corruption, relatively efficient bureaucratic instititutions, and higher levels of democratic accountability seem to increase the likelihood that World Bank projects will succeed. In fact, socioeconomic conditions seem nearly as important as QOS to a project’s final outcome rating.
Of course, these very simple regressions do not constitute definitive results. But our preliminary findings paint a picture familiar to many development practitioners: while supervision of aid projects is important, so too is understanding how general contextual factors shape the success and failure of projects.