November 2, 2006 | Overseas Development Institute
The African Green Revolution and the Millennium Villages Project
Speaker: Pedro Sanchez, Director, Tropical Agriculture and Rural Environment Program Director, Millennium Villages Project, The Earth Institute at Columbia University | An ODI Event

Monitoring and Evaluation in the Millennium Villages Project

The Millennium Villages project is underpinned by a robust monitoring and evaluation platform using longitudinal survey systems in all 14 cluster sites and 10 countries to help determine the impact and effectiveness of the intervention work in health, education, agriculture and infrastructure.  Through this, project practitioners employ learned experiences to: improve the planning and allocating of resources; strengthen service delivery; and ultimately demonstrate key results to policymakers, governments and stakeholders. 

Below are the four components to the Millennium Villages evaluation: impact assessment, performance monitoring, process evaluation and cost.

IMPACT ASSESSMENT

The initial choice of MVP sites was purposive and non-random. While the specific interventions contained within the MVP package have scientifically proven efficacy, assessing the adequacy and impact of the delivery system is imperative given the pressing challenges to health and development in much of sub-Saharan Africa.  Sites were chosen to represent a diversity of major agro-ecological zones that together represent the farming systems used by 90% of the agricultural population and 93% of the agricultural land area of sub-Saharan Africa.  A criterion for MVP site selection within each agro-ecological zone was that villages were 'hunger hot-spots', as identified through the UN Millennium Hunger Task Force, indicating at least 20% prevalence of underweight children under the age of five. 

The impact assessment uses standard agronomic research techniques and a pair-matched community intervention trial design with cluster-level analysis. In agriculture, a random sample of MVP staple crop plot yields is compared to a random sample of control plots in each program site for each year’s major growing season.  In addition, a detailed household mapping was conducted within each Millennium Village cluster prior to the start of interventions. This process included a census, GPS coordinates for most dwellings, and a household wealth score. Three hundred wealth-stratified randomly selected households were selected at random to participate in each of the three assessment rounds -- at baseline, year 3, and year 5.  Year 3 mid-term results are scheduled for publication in 2010.

Comparison villages have been selected for inclusion in the impact assessment in 10 MV sites during year 3 assessments. These villages are randomly selected from three potential candidates that are "matched" to the MVs on a number of baseline characteristics relevant to the MDGs. These include village size, accessibility, agro-ecological zone, principal livelihood strategies, crop types, electrification, the presence of clinics and schools, and other partners working in the area. The inclusion of comparison villages at this stage was felt to be important to assess "causality" -  whether observed changes in the MDG outcomes were a result of exposure to the Millennium Villages Project as opposed to wider social forces unrelated to the intervention. This dimension of the work is critically important given the 5-10 year study horizon and the rapid pace of change currently underway in many parts of sub-Saharan Africa.

• Household Surveys: are administered to all household heads and collect information on household demography, education, employment, agricultural and non-agricultural sources of income, assets, expenditure, consumption and access to basic services including water and sanitation, and energy, transport and communication.
• Adult Surveys: are administered to all adults aged 13-49 years old to assess health-related MDGs, nutrition and food security, alongside common causes of child mortality including diarrhea, pneumonia and malaria, and health seeking behavior.
• Biological testing: tests for malaria (thick and thin smears) and anemia using a Hemocue point-of-care device is conducted among all under-5s
• Anthropometric data: weight, height and mid-upper arm circumference is assessed among all under 5s using standard protocols

The overall impact assessment will adhere to guidelines in Transparent Reporting of Evaluations with Nonrandomized Designs (TREND). This includes employing a clear theory-driven conceptual framework to guide intervention design, delivery and evaluation; the description of the settings and state of service delivery in both intervention and comparison communities; documentation of the intervention package at each site; monitoring and reporting on exposure and uptake of interventions; and the presentation of data on a full-range of pre-defined study outcomes using a pre-specified analysis plan.

PERFORMANCE MONITORING

In addition to surveys, data are also collected across the Millennium Village sites on a monthly and quarterly basis, depending upon the indicator. A quarterly report system of performance monitoring indicators from 9 different sectors provides a standardized approach to documenting the timing, sequence, and uptake of interventions and help to define the local contents, key activities and initial outputs of the MVP package.  

PROCESS EVALUATION

Understanding the process of implementation is particularly relevant to the MVP. While the individual components of the package are of proven value, the systems necessary to support their integrated delivery in a diversity of settings are poorly understood. To address this, a portfolio of implementation science (or process evaluation) activities will be conducted alongside the project. The work is largely qualitative and will detail the full inventory of interventions undertaken; document their timing and sequence; describe how the model was adapted to the local context; how implementation differed from original program planning; highlight barriers and facilitators to implementation; examine cross-sectoral synergies; and unpack what worked well, what didn't, and why.

COST

The MVP is based on the assumption that rapid progress towards the MDG targets can be attained at a cost of $120 per person per year, sustained over a 5-10 year period.  The MVP directly finances half of this targeted scope of activities with a budget of $60 per person per year. These budget models are derived from a detailed bottom-up needs assessment generated by the UN Millennium Project and WHO Commission on Macroeconomics and Health, and are entirely feasible within the G8’s promised level of development assistance to Africa by 2010.

There are several sources of inputs into Millennium Villages - inputs through core project partners (Columbia University, Millennium Promise, UNDP,), support from government, external partner organizations such as NGOs, material resources from the community, and in-kind support. The nature and intensity of these inputs is likely to differ substantially between sites.

An economic costing study has been introduced to monitoring project inputs. The aim of the assessment is to document the absolute and relative contribution of project partners to all cluster-level activities within the Millennium Villages that contribute towards achieving the MDGs.