Does this sound like a familiar evaluation case to you? The project or program to be evaluated is built around a rigid logframe approach but you know that the reality on the ground is more complex. The assignment’s terms of reference ask for a quick review with limited time and budget. In this situation, how to add value to the client and innovation to your services as an evaluator? We tested the Outcome Harvesting approach in the final evaluation of a conservation fund in Tanzania in November 2016. Read along to find out the lessons we learnt.
Eastern Arc Mountains are a critical source of water and biodiversity for the surrounding areas. (Photo: EAMCEF)
The assignment was to evaluate the project “Improving conservation of the Eastern Arc Mountains forests of Tanzania through The Eastern Arc Mountains Conservation Endowment Fund (EAMCEF)” funded by Norway. The main goal of the project was to enhance conservation of forests by:
- Providing alternative livelihoods to local people;
- Strengthening protected area management; and
- Supporting research activities.
Our expectation was that evaluating the project against the existing logframe only would not allow us to understand sufficiently the local dynamics. We wanted to know what has happened after the beneficiaries have received incentives (such as agricultural inputs) for reducing pressure on the protected areas. For example, if farmers receive materials for beekeeping, how does it change their behaviour in terms of using the natural resources inside the reserves? What other consequences does the intervention cause in the village? Or in other locations linked to the same landscape?
Outcome Harvesting has recently received increased attention among development evaluators
Outcome Harvesting is a method that allows to systematically record the changes that occur in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organisation, or institution” (Wilson-Grau 2012).
The approach helps to find out trends of positive and negative, expected and unexpected consequences of the intervention by using a simple form (outcome card) that is adapted to each specific case. Evaluators or project M&E staff, together with key stakeholders (so called change agents), fill in the information. Subsequently, the information is verified from other sources to validate the outcome. If the situation allows, dozens or even hundreds or outcome cards can be collected. The data can be used for simple but interesting statistical analyses and visualisations.
High visibility has been given to the approach in recent evaluation conferences, e.g. in the European Evaluation Society Biennial Conference 2016 and the American Evaluation Association Conference 2016.
We invited a selected group of beneficiaries to an Outcome Harvesting workshop to discuss the positive, negative, expected and unexpected effects of the project. As the next step, the evaluation team carried out a field verification mission to the participating villages. (Photo: Petra Mikkolainen)
Outcome Harvesting is potentially time-consuming and expensive (see also Sainsbury, K. et al. 2015). Therefore, it is challenging to include it in short mid-term reviews or final evaluations. We used the methodology for specific case studies as part of the overall evaluation.
The added value of Outcome Harvesting in the final evaluation of the Fund was that we could demonstrate how different project activities had interactions that were not captured in the project documents and reports. For example, downstream beneficiaries of improved cooking stoves were already benefitting from the nearby watershed for irrigation. Upstream, the communities who were expected to protect the forest felt resentful because they considered that the benefit-sharing system was not fair.
We concluded that Outcome Harvesting is an option when implementing organisations have enough openness and curiosity to learn from both positive and negative experiences. It also works better when the M&E activities can be carried out in parallel with the project execution. For short-term evaluation assignments, lighter methods such as the Most Significant Change offer similar added value as Outcome Harvesting for capturing stakeholder experiences.
More information from Outcome Harvesting can be obtained from here.
Wilson-Grau (2012) Outcome Harvesting. The Ford Foundation.
Sainsbury, K., Burgess, N. D., Sabuni, F., Howe, C., Puis, E., Killenga, R., & Milner-Gulland, E. J. (2015). Exploring stakeholder perceptions of conservation outcomes from alternative income generating activities in Tanzanian villages adjacent to Eastern Arc Mountain forests. Biological Conservation, 191, 20-28.