Impact Evaluation of the SADA Millennium Villages Project in Northern Ghana

The Millennium Villages Project (MVP) aimed to demonstrate how the Millennium Development Goals (MDGs) could be achieved locally through an integrated approach to sustainable development.

While the MDGs have now been superseded by the Sustainable Development Goals (SDGs, 2016–30), there remains a consistent thread to the MDGs around the important global goals of eradicating poverty, preventing avoidable deaths, improving education and so on.

It is hoped that the evidence and analysis will be of relevance to a wide range of actors in international development.

The MVP was first piloted in Kenya and Ethiopia, and in 2006 was launched at scale. In 2012, the UK Department for International Development (DFID) funded £11 million into an MVP in Northern Ghana that ran until December 2016.

The project targeted a cluster of communities of up to 26,500 people in the West Mamprusi, Mamprugu Moagduri and Builsa South districts of Northern Ghana – an extremely poor area with 80–90% of the population living below the national poverty line.

MVP implementing partners

The project

Timeline

2012 – 2016

Funding

£11 million

Area of evaluation

Northern Ghana
Northern Region

West Mamprusi District
Mamprugu Moagduri District

Upper East Region
Builsa South District

26,500 people in a cluster of villages

80-90% of people living below the poverty line

Evaluation Design

The Millennium Villages in Northern Ghana Impact Evaluation (MVE) is a fully independent, robust evaluation that aims to provide evidence of the impact, sustainability and cost-effectiveness of the MVP model, and help inform future rural poverty reduction interventions.

5 years duration

5 survey rounds of households and repeated interviews

Housholds surveys in 35 villages within the site and 68 control villages

Data collection on a wide range of welfare indicators

Data collected from sizeable project and control groups

Confirmatory and exploratory analysis

3 qualitative impact assessments

The evaluation team

Difference-in-difference design

The mixed method evaluation used a difference-in-difference (DiD) design at its core. This approach compared changes in outcomes in the MVP areas with changes in the same outcomes for an explicit comparison group. The validity of a DID approach rests on the assumption that project and comparison groups are similar.

At baseline, the comparison groups were chosen by matching district villages to project villages using a propensity score built through village-level characteristics obtained from census data and field visits.
To remove baseline differences in characteristics between the project and comparison groups, the evaluation employed matching methods at household and individual levels.

Quantitative data collection

Sample survey of
755 households in
project villages

1,496 households
in comparison
villages

These sample sizes were able to detect impacts of an acceptable size though power calculations. The size of the comparison group is twice the size of the project group to:

stratify the impact of the intervention by distance in order to identify spillover effects; and

perform matching of observations at the household level to further improve the comparability of the two samples.

In every survey round, the same households were targeted for interview, though at each round not all targeted households were found. Attrition rates were very low: less than 8% of the original target sample of households was lost over time and over 85% of the original target individuals were included in the last survey round.

Survey Instruments

Qualitative studies

There were three qualitative modules

An institutional assessment that captures institutional change, particularly at the district and community levels

A reality check approach that uses a condensed immersion approach to better understand the realities of households and unintended consequences for them

An ‘interpretational lens’ that uses an adapted form of Participatory Rural Appraisal to obtain feedback and insights on preliminary statistical findings from the perspective of different well being groups

Key evaluation findings

The evaluation found that the project site in Northern Ghana produced a significant impact on seven of the 28 MDG outcome indicators, but did not meet its stated goals of achieving the MDGs by 2016.
It led to positive impacts in areas of education and health. Primary school attendance increased by 7.7% and the number of births attended by skilled professionals increased, as did the use of contraception and the number of children sleeping under mosquito bed nets. There was also a large impact on access to improved toilet facilities.
The evaluation observed no impact of the project on the level of poverty or hunger. It also did not improve some of the outcomes it explicitly targeted, such as child mortality, immunisation rates, antenatal care, access to drinking water and usage of mobile phones.
Learning from the evaluation indicates that rigorous evidence must be programmed in at the earliest opportunity to inform significant investments such as the MVP to ensure positive impact and cost-effectiveness.
All of the evaluation’s data is available as open data and is intended to be used by researchers in Ghana to benefit any future studies and projects in the region.

Impact on poverty

In Northern Ghana, the MVP did not have an impact on the indicators of eradicating extreme poverty and hunger, with the exception of reducing poverty measured using household income data and adjusted by purchasing power parity.

It did not, however, achieve the MDG target to reduce extreme poverty and hunger at the local level (Goal 1), whether measured by the national poverty line or the national food poverty line. There has been no impact on the percentage of undernourished children.

Publication: MVP Briefing 1: What did it achieve?

Cost effectiveness

Health and infrastructure were the largest sectors in terms of project spend, with management and overheads accounting for around a third of the total. The cost per capita was US$360 in 2012 present value terms, or US$88 per capita per annum.

Across the key impact areas of income, health and education, the MVP compares unfavourably with other projects in terms of the cost-effectiveness of service delivery. The returns to investment in education appear to be highest, although it is believed that similar results could have been achieved at significantly lower cost.

Sensitivity analysis shows that transferring such a project to local ownership could improve the value-for-money proposition, but that even a 50% cut in overheads would still yield questionable cost-effectiveness overall.

Cautionary note

Paper No 3: Was it cost-effective?

Systematic reviews

Sustainability of the Millennium Villages Project

The MVP aimed to strengthen local institutions and community capacity to ensure that project gains were sustainable.

According to district officials, the MVP created a resource-intensive parallel structure. Project management was not integrated or embedded into existing decentralised government structures, and the MVP team continued to manage the bulk of funds and account to the donor directly.
Many of the ‘free service’ interventions that the MVP provided (such as the supply of medicine through the free National Health Insurance Scheme) lacked ongoing funding.
The evaluation concluded that the exit strategy and sustainability of MVP interventions was not fully considered at the design and planning stage.