CEval and Indufor presented the results of the recent meta-evaluation on Finland’s development cooperation on Thursday 15th March 2018. The quality of the evaluation report was praised by the Ministry for Foreign Affairs of Finland (MFA). However, the results turned out to be rather concerning. The MFA is now faced with a decision on how to ensure sufficient quality of the so-called decentralised evaluations (i.e. assignments managed by the regional units, not by the Development Evaluation Unit) so that they truly contribute to learning.
MFA staff thanked the evaluation team for developing an excellent meta-evaluation tool that can be used also in the future meta-evaluations. From the left to right: Riitta Oksanen, Deputy Director of the Department for Development Policy; Kristiina Kuvaja, Department for Africa and the Middle East; Susanne Väth, Evaluation Deputy Team Leader (CEval); Stefan Silvestrini (CEval), Team Leader; Petra Mikkolainen (Indufor). Photo: Jussi Viding.
The main conclusion of the evaluation was: “While we cannot conclude on the adequacy of MFA’s decentralised evaluation portfolio, we can conclude on the quality of the terms of reference and the quality assurance of evaluation reports: both leave room for improvement which in turn reveals capacity gaps within MFA.” In other words, there are some serious challenges in the quality of decentralised evaluations in Finnish development cooperation. On the other hand, our CEval colleagues also reminded that Finland did not do worse compared to many other meta-evaluations that they have carried out in the past.
Nevertheless, what could be the way forward? Eliminate evaluations that are in the pipeline or at least some of them? Maybe group them into bigger assignments? Delegate them to the Development Evaluation Unit who are experts in this topic? Or automate the evaluation processes with better templates and guidelines?
These are some of the questions that MFA is to reflect internally when it will follow up on the recommendations of the evaluation.
The meta-evaluation’s results in a nutshell:
This meta-evaluation covered 51 decentralised evaluation reports and 45 corresponding terms of reference developed between September 2015 and August 2017. The documents comprised 23 mid-term and 28 final evaluations of bilateral, multilateral or multi-bi projects and programmes commissioned by various thematic units and regional departments of the MFA, embassies and multilateral partners.
The assignment consisted of two parts: (i) a meta-evaluation assessing the quality of the evaluation reports under consideration, and (ii) a summative meta-analysis aggregating the contents of these reports.
The following key recommendations were derived from the analyses:
- Run an inventory of all interventions classified by key characteristic to enable stratified sampling in the future.
- Improve the evaluation manual substantially, to enhance knowledge of evaluation methodologies and practices and to consider improving existing structures, e.g. via stronger centralisation of the knowledge management system and better coordination with the Development Evaluation Unit (EVA-11).
- Enhance quality assurance and to ensure methodological expertise when recruiting evaluators.
- Ensure that the results of this meta-evaluation are sufficiently disseminated to feed back this information to implementers.
The team was led by the German consortium Particip supported by another German institution specialised in evaluations, CEval. The Ministry for Foreign Affairs of Finland (MFA) commissions meta-evaluations on a regular basis. This is done through the Development Evaluation Unit, an independent body, through a long-term Evaluation Management Services contract with Particip and Indufor.
The evaluation report and webinar will be published shortly in the MFA website.
To obtain further information on Indufor evaluation services, please contact petra.mikkolainen (at) induforgroup.com.