I EL MELÓLOGO.

Evaluations often originated due to a need for accountability as demonstrated by the 1960s US Government investments in social programmes and the Rwandan Genocide. The Development Assistance Committee defines accountability as an

“Obligation to demonstrate that work has been conducted in compliance with agreed rules and/or plans. This may require a careful, even legally defensible, demonstration that the work is consistent with the contract terms” (DAC, 2002, p. 15).

In humanitarian evaluations, programme managers, staff, decision makers and politicians are accountable to communities, citizens, service users, tax payers, advocacy groups, relevant professions, international organisations and donors (OECD/DAC, 1991; Rogers, 2005). According to Rogers (2005), the common form of accountability focuses on meeting targets, outcomes or outputs. Discrepancies are reported between targets and performance to funders with the assumption the information will inform subsequent policy decisions.

The very definition and meaning of evaluating humanitarian assistance creates what Cracknell (2000) refers to as the ‘accountability/lesson learning’ dichotomy. Cracknell and other authors have highlighted there is an essential tension between these two objectives. Accountability focuses on questions relating to whether humanitarian assistance works or is effective and accountability related concerns in evaluations underlie the interest of donors, governments and the media – all of whom have a justifiable interest in ensuring that aid money is spent effectively and responsibly. Lesson learning however, is focused on understanding why particular activities are more or less successful to improve the effectiveness of future humanitarian relief activities and as such it requires a different evaluation approach (ALNAP, 2001).

This tension is significant because evaluations designed to meet one objective will be of limited use in satisfying the other. In deciding upon a framework for evaluations in humanitarian crises, it is thus necessary to decide which objective is of primary interest, or to accept that different evaluations can and should have a different focus with respect to accountability and lesson learning. In either case, it is necessary to be clear from the outset when designing an evaluation which objective is of primary importance (Poletti, 2004). Patton (1997) stresses who an evaluation is for, and how an evaluation will be used should drive the entire evaluation process from the initial commissioning, through to the execution of the evaluation itself, the analysis and report writing stage, and to the dissemination of the report and subsequent follow-up. Agencies involved in the delivery of

humanitarian relief and indeed this research are focused on lesson learning, as it is vital to improve performance and increase the impact of humanitarian activities.

“Experience reflected upon is the handmaiden of progress. Evaluation is an integral part of individual and institutional learning. By doing, evaluating and doing again we learn to do better” (Jean Quesnel, Chairman of the DAC Expert Group on Aid Evaluation, 1993, p. 3).

Terms such as ‘lessons learned’ and ‘informed judgements’ are common themes among the definitions in Table 2, which denotes the role of evaluation in providing lessons for future programmes through feedback (OECD/DAC, 1991). The Development Assistance Committee defines lessons learned as

“Generalizations based on evaluation experiences with projects, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact” (DAC, 2002, p. 26).

A key aspect is absent from this definition – there is a focus on the attributes of the ‘lesson’ but not about the process by which it may, eventually, be used. In this sense, they are ‘lessons learned from experience’, as opposed to ‘lessons applied’. Most evaluation professionals would probably acknowledge that their lessons are under-utilised. Spilsbury et al. (2007) highlights two key problems underlying this observation: 1) lessons are poorly formulated (low quality); and 2) processes to promote dissemination and uptake of lessons are weak.

A high quality lesson should; succinctly specify the context from which it is derived, establish its relevance beyond that context (where it will be applied and by whom) and suggest some prescription or action. Although lessons are derived from a specific situation, they are intended to have wider relevance. Lessons however, are often regarded as one-off findings that lack supporting

information from other sources (Spilsbury, et al. 2007). Lessons that are supported by triangulated evidence command greater credibility among their potential users.

“The more rigorous the supporting evidence, and the greater the triangulation of supporting sources, the more confidence one has in the significance and meaningfulness of a lesson learned” (Patton and Millet 1998, p. 14).

Clearly, producing high-quality lessons is necessary but not sufficient to maximise their potential utility. Relying on passive dissemination approaches, for example, by simple dissemination of evaluation reports, is a common but not very effective method of promoting their uptake. As a result, many lessons are destined to be archived in under utilised databases or to languish, unheeded in evaluation reports. Greater emphasis on enhancing the credibility and building the ‘ownership’ of lessons is required. A variety of complementary communication and ‘outreach’ processes are needed to enhance the uptake of lessons by their intended users (Spilsbury et al. 2007).

When conducting a lesson learning evaluation it is essential to understand how lessons learned will be used and the extent to which policies and programmes can be influenced. The evaluator and commissioning agency must have a mutual understanding of how lessons learned will be used to drive and focus the evaluation but also to ensure the evaluation results are used effectively. The researcher undertook lengthy discussions with each of the commissioning agencies for this study to ensure seeking and utilising lessons learned were a key objective of each agency; to understand how lessons learned would be used; and how lessons would be fed back to recipients or beneficiaries of the interventions at the local level.

Lessons learned tend to be policy, programme or project oriented in the form of ‘recommendations’. Hence, the focus is often in building the capacity of donors and humanitarian agencies in designing policies and programmes whilst the capacity, resilience or vulnerability of beneficiaries is of peripheral importance, if not of little relevance (Kaiser, 2002 and Hallam, 1998). The

objectives of humanitarian evaluation have hitherto related predominantly to institutional priorities. According to Kaiser (2002) there has been no consideration that beneficiaries might have a role other than as recipients of improved assistance or that there might be value in the evaluation process for beneficiary populations. Involving beneficiaries in the evaluation of humanitarian assistance programmes implies that evaluation objectives are wider than a straightforward attempt to measure programme outputs. Any meaningful evaluation of assistance programmes requires analysis of both the socio-political economy inhabited by those affected by complex emergencies and the survival strategies they employ. Without beneficiary input, evaluation becomes counterproductive (ibid.).

Evaluation purpose ranges from lessons learnt for the implementing agency to accountability, including fiscal accountability to the donor. These might be referred to as evaluations for internal purposes and evaluations for external purposes. In the uptake of evaluation especially evaluative capacity within organisations, the tendency has been to establish a purpose that is internal i.e. lessons learnt rather than external accountability.