We’d like to thank Wanjala and Muradian for their paper on agricultural productivity and income in the Sauri Millennium Village (MV). The Sauri site was initiated in 2005-6 as the first MV field site in sub-Saharan Africa. Through employing data collected at a single household visit in 2009 among Sauri MV and households from nearby villages, the authors make statements about the impact of the project by comparing agricultural productivity and income between these two groups. We have a number of scientific concerns regarding this paper:
1) Problems with comparison villages: The authors’ findings hinge on comparing Sauri to several villages in the surrounding area. In order to make this a valid comparison, it is essential to know whether these villages and households were similar at baseline as well as what took place during the project period.
The first problem is that these villages are not comparable. The authors own paper acknowledges that villages were significantly different for 7/10 non-outcome characteristics assessed – including age, education, household size, dependency ratio, type of house, livelihood, and size of land holding (Table 3 of their manuscript). They attempted to address these differences by incorrectly applying a statistical technique called ‘propensity score matching’ (PSM) - which matches each household from the MV site with an ”equivalent” household from a non-MV site based on the characteristics highlighted above. There is an extensive literature emphasizing this technique should not be undertaken unless individuals can be matched using baseline data on the outcome of interest (see Cook et al. [1] and Glazerman et al. [2] for more on this important point). As the authors collected data only after the intervention began, their analysis is unable to meet this very important condition.
The second problem relates to understanding what took place in these villages during the Project period. For the authors’ analysis to hold up, these villages should be ‘unexposed’ to the intervention. However in 2008, over 2500 households in the villages of Siriwo and Wagai (two of the five non-intervention villages) benefitted from the distribution of 160 tons of fertilizer and 22 tons of seeds facilitated by the Sauri MV, valued at 9.1 million Kenyan Shillings (roughly $125,000). These were purchased by the Constituency Development Fund (CDF) and distributed by the MV Project. The result of this ‘spillover’ would reduce differences between MV and non-MV groups in their agricultural production and income, and understate the true impact of the intervention on the main outcomes of productivity and income.
2) Measurement problems in assessing agricultural income: Assessing household income is notoriously difficult, particularly in low-income rural areas where livelihood strategies are complex. The calculations of agricultural productivity and income used by Wanjala only considered crops sold on the market and not total production (i.e. crops grown and consumed, stored, milled/processed or exchanged for barter). This substantially undervalues agricultural productivity and income in this area of western Kenya that relies heavily on subsistence farming for ‘own-consumption’ rather than sales.
3) A basic misunderstanding of the Project: It’s important that any effort to evaluate the Millennium Villages be aligned to the central objectives laid out by the project A recent Project document cited by the authors [3] notes “the first years of the MVP (years 1-3) focused on increasing staple crop production to increase food security (p 21)” – a strategy critical for addressing persistent hunger and undernutrition.
Evidence from the projects’ peer-reviewed scientific publications suggests over the first three years staple crop yields have increased two to three fold [4] with parallel improvements in food security and diet diversity. Sauri alone has witnessed major reductions in chronic undernutrition - measured by levels of stunting (growth failure) among children under 2 years old. At baseline 62% of children were stunted, with levels reducing to 38% after three years [5]. Program-wide across nine sub-Saharan African sites an unprecedented 43% reduction in chronic undernutrition has been observed over the first three years of the project [6].
4) Gains in household income: Finally, preliminary data from Sauri cluster provides evidence of substantial economic gains after the project’s first five years. Household income has more than doubled, with about 70% of these gains were due to increased agricultural income. While income from maize production has increased five-fold since baseline, more than 2/3 of agricultural income gains are from non-maize activities including crop and livestock diversification, and value-chain interventions such as agro-processing. Major efforts underway by the site include:
• Diversification to high value crops including tomatoes, onions, watermelons, and tissue culture bananas. In the past year, a total of 2,295 farmers participated in these enterprises.
• Greenhouses: The site has established 42 functional green houses to better support the year-round production of high value crops
• Fish farming: In 2011, the 435 fish ponds were under operation yielding 5,322 tons of fish for farmers
• Livestock: Including dairy goat rearing and poultry farming. • Bee-keeping: The site has trained and supported 260 bee keepers who manage 930 hives at 33 communal sites. In 2011 these famers harvested 868 kgs of honey valued at over 100,000 Kenyan Shillings.
• Farmer cooperatives: The site has found that through working together, farmers can be more profitable. As a result, six cooperatives have been established to support production and marketing of cereals , honey, horticulture, poultry, dairy and fish.
In summary, the Wanjala and Muradian paper has serious technical flaws, and provides little insight into some of the transformative changes that are actually taking place in the Sauri MV cluster. To our knowledge, while their work has been posted on their internal website, it has not yet been published. While we thank the authors for their interest in the project, we suggest their conclusions are premature.
References
Cook, T.D., W.R. Shadish, and V.C. Wong, Three conditions under which experiments and observational studies produce comparable causal estimates: new findings from within-study comparisons. Journal of Policy Analysis and Management, 2008. 27(4): p. 724-750.
Glazerman, S., D.M. Levy, and D. Myers, Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy of Political and Social Science, 2003. 589: p. 63-93.
The-Earth-Institute and Millennium-Promise, Harvests of Development in rural Africa: the Millennium Villages after three years. 2010, The Earth Institute, Columbia University: New York.
Nziguheba, G., et al., The African Green Revolution: Results from the Millennium Villages Project. Advances in Agronomy, 2010. 109: p. 75-115.
Fanzo, J., et al., A 3 year cohort study to assess the impact of an integrated food- and livelihood-based model on undernutrition in rural western Kenya. Food-based Approaches for Combating Micronutrient Deficiencies, ed. B. Thompson and L. Amoroso. 2010, Rome The Food and Agriculture Organization of the United Nations and CAB International.
Remans, R., et al., A multi-sector intervention to accelerate reductions in child stunting: an observational study from nine sub-Saharan African countries. American Journal of Clinical Nutrition, 2011. doi: 10.3945/ajcn.111.020099.
