OECD-DAC evaluation criteria
The OECD-DAC criteria are the minimum standard for evaluating development interventions. An evaluation built on them assesses:
- relevance — whether the intervention is doing the right things for its beneficiaries and context;
- coherence — how well it fits with other interventions and policies;
- effectiveness — whether objectives and intended results were achieved, and what unintended results emerged;
- efficiency — how well inputs, time and money were converted into results;
- impact — the broader, longer-term changes the intervention contributed to;
- sustainability — the likelihood that benefits continue after external support ends, seen from financial, social and institutional perspectives.
Alongside the criteria we examine the quality of the intervention’s concept and design, the quality and timeliness of implementation and monitoring, and the involvement of stakeholders in the project’s processes and activities.
Logical Framework Approach (LFA)
The Logical Framework Approach is a management tool used mainly in the design, monitoring and evaluation of international development projects; it is also known as Goal-Oriented or Objectives-Oriented Project Planning (GOPP / OOPP). Used as its authors intended, LFA imposes a logical discipline on the design team, and the result is a high-quality project design.
The logical framework itself is a four-by-four matrix. The rows describe four types of events during implementation — activities, outputs, purpose and goal. The columns provide four types of information about each of them: a narrative description; objectively verifiable indicators; means of verification; and assumptions — the external factors that may influence the events but cannot be controlled by project management. A good design substantiates its assumptions, especially the potential “killer assumptions” whose failure would have major negative consequences for the project.
The core of the matrix is a chain of connected hypotheses:
- if these activities are implemented and these assumptions hold, then these outputs will be delivered;
- if these outputs are delivered and these assumptions hold, then this purpose will be achieved;
- if this purpose is achieved and these assumptions hold, then this goal will be achieved.
IMACON partners authored a Russian-language guide to LFA and a Belarusian translation of the classic NORAD planning guide — both available in Publications.
Project Cycle Management (PCM)
Project-management methodologies define standard processes that improve the quality of the project life cycle — its relevance, feasibility and effectiveness, including how well the project is managed. All of them follow a flow similar to the plan–do–check–act cycle, with phases linked by deliverables: the result of one phase (a document, a report, an approval) becomes the input of the next.
Several agencies — the Asian Development Bank, the World Bank, the European Commission and national governments — have adopted their own versions. IMACON works primarily with the European Commission’s Project Cycle Management manual (2004), adopted by many development partners as the standard for project development, funding and evaluation. The EC project cycle has five phases: programming, identification, formulation, implementation and evaluation.
Institutional Development / Organisational Strengthening (ID/OS) and the Integrated Organisational Model (IOM)
ID/OS and IOM are instruments developed by our partner MDF in the Netherlands.
ID/OS starts from the idea that development interventions succeed only with broad-based support: those who manage and evaluate them, those responsible for implementation and the intended beneficiaries all take part in planning. Development results are sustainable when organisations can maintain an appropriate level of service long after major financial, managerial and technical assistance from external donors has ended. ID/OS tools help assess and design interventions in terms of the institutional capacity of the organisations involved — a process that requires patience, time and a flexible working relationship rather than a single short-term project.
IOM is a model for describing, analysing and diagnosing how an organisation functions. It pays particular attention to the relationships between the elements of an organisation: they can be considered separately, but they are interconnected and should ideally be balanced. Where the balance is missing or unclear, the organisation cannot work at full effectiveness and the need for change becomes evident. IOM is universal — it applies equally to a government department, an NGO, a local authority or a private company — but it is an organisational concept rather than a single tool, and deeper analysis calls for more specific instruments chosen for the purpose.
Outcome Mapping
Outcome Mapping, developed by Canada’s International Development Research Centre (IDRC), focuses on outcomes understood as changes in the behaviour, relationships and activities of the people, groups and organisations a programme works with directly. Through outcome monitoring an organisation can claim contributions to outcomes rather than merely the delivery of outputs. The method:
- defines a programme’s outcomes as observable changes in the behaviour of its direct partners;
- focuses on expected significant changes and on how the programme facilitates change rather than controls or causes it;
- recognises the complexity of development processes and the contexts in which they occur;
- looks at the logical links between interventions and outcomes instead of attributing results to any single intervention;
- requires the involvement of programme staff and partners at all planning, monitoring and evaluation stages.
Most Significant Change (MSC)
Most Significant Change (Davies and Dart, 2004) is a participatory monitoring and evaluation method. Stakeholders decide which kinds of change should be recorded and take part in analysing the data. Stories of significant change are collected at field level, and the most important ones are selected systematically by panels of designated stakeholders or staff. Its advantages:
- participants choose what information to collect, and diverse rather than standardised data is used;
- information is analysed by all participants, not only by a central unit, and subjectivity is used rather than avoided;
- stories are a rich way to express the changes happening in people’s lives, free from pre-set indicators;
- the method is non-linear and open to the outcomes participants themselves see in their lives;
- it orders a large amount of information into an overview of the most significant changes brought about by an intervention.
Organisational Analysis (Forum Syd)
This methodology of organisational analysis within not-for-profit organisations was compiled by the Swedish organisation Forum Syd, largely from Sida material assembled by consultant Peter Winai together with internationally active non-profits, and later revised by Greger Hjelm of Rörelse & Utveckling, Stockholm.
It brings to the fore different perspectives on organisational development and the process of change — including the roles an external consultant may adopt when assisting with analysis and development. It offers working models (the staircase model, the INTRAC model and others) for describing and assessing an organisation, and its main practical tool is a series of questions to use when carrying out an organisational analysis. The method introduces a way of working with analysis and development; it does not claim to provide ready-made formulas for success.
Octagon (Sida)
In 1999 Sida invited a number of Swedish NGOs to discuss how the results of support for capacity and organisational development in local partner organisations could be measured. In May 2000 Peter Winai presented the Octagon — a model for assessing and following up organisations — to this group on Sida’s behalf.
The Octagon is a tool for assessing strengths and weaknesses in NGOs. It can structure the dialogue with a partner organisation, support the selection of partners, group partners by their needs for internal development, or identify the moment when a funder should phase out its support. It builds a comprehensive picture of an organisation’s capacity and development profile by systematically reviewing four aspects:
- the organisation’s objectives and its management and administrative structures — the organisational base;
- its activities with or for its target groups — the output;
- its capacity to succeed in its work: professional skills, funds and administrative systems;
- its capacity to create and maintain relations with target groups and other actors in civil society.
Repeated systematically over several years, the analysis tracks change in the organisation — which makes the Octagon suitable both for baseline studies and for measuring the results of internal organisational development over time.
SenseMaker
SenseMaker links micro-narratives with human sense-making to create decision-support, research and monitoring capability in organisations large and small. It brings a quantitative approach to what has traditionally been a qualitative domain: large numbers of short stories, indexed by their own authors, reveal patterns, weak signals and the influence of culture on decision making. SenseMaker supports evidence-based policy under conditions of uncertainty and the engagement of whole workforces or communities in monitoring.