Defining monitoring and evaluation

How do we Evaluate Programmes?

Definitions

Attention: do not confuse project monitoring with project evaluation. 

Project monitoring is identical with project performance measurement and belongs to the project execution and control  phase of project management. Monitoring and reporting the project performance is to review the project progress against expected milestones, timelines and costs. The purpouse of reporting is to share the information required to manage  Manage CSSQ (Cost, Scope, Schedule, and Quality). . With reference to tasks, actions and projects, monitoring is looking at what is currently happening. It is functional to the objective of delivering the project outputs with the expected quality and within the time and cost constrains defined in the project plan document. The deliverable of project appraisal is the Project Status (or progress) report (see project execution templates).

Project evaluation is not a part of project execution but is a different phase of the programme cycle Project evaluation is not done by the project manager but by the project sponsor (e.g. the Programme Manager, etc.) or by someone supporting her. It requires Project Status reports as a fundamental input. Its outputs are projects evaluation reports  and program lessons learned. Evaluation is based on monitoring but adds a "judgment" on the correlation between activities performed, outputs delivered, changes induced objectives achieved and impacts obtained upon the factors generating the problems and opportunities that motivated the project. While the delivery of project outputs is the responsibility of the project team, the achievements of project outcomes and objectives depends also by the way stakeholders utilize the project outputs to interact with the rest of the community and contribute to achieve the project objectives.

Monitoring is looking at what is going on. With reference to tasks, actions and projects, monitoring is looking at what is currently happening. for example: Are the people working at their tasks? Are these tasks being executed? Are the expected deliverables being delivered? Is the money being spent?  Are the procurements being consigned to the team? Are the outputs consigned to the beneficiaries?.  Through monitoring we arrive to an "assessment of facts".

As we have mentioned elsewhere, in a projectised organisation, programmes are implemented through projects. And in order to implement projects, teams carry out activities and tasks. Before beginning a project, the project manager, along with his team, makes a work plan that details the timelines of activities, individual responsibilities in the team and budget allocations for each activity. The work plan helps the team in having clarity on what they are doing, on their timelines to complete activities and tasks and on what their budgets allocations are. If you are a project manager, you need the workplan to monitor whether your team is implementing the project as planned and is spending within stipulated budgets. Monitoring is thus an integral activity of project management and implementation. The activity of establishing a monitoring and evaluation system is an activity in itself and must be planned well.

In order to make corrective measures, it is important that the information emerging from monitoring is analysed at the end of each project year. This exercise of analysing monitoring information is called a monitoring review. The objective of monitoring review is to empower staff in delivering programmes effectively. The learning that emerges from a monitoring review must be integrated into current programme implementation. In order that the learning become integrated into new programming, the information must be shared, among the wider programme team, during country learning reviews usually held every alternate year.

Evaluation is based on monitoring but adds a "judgment".  Is what happening supposed to happen? What we are doing is it the right thing? What we have done, was it done right?  Do the actions follow the plan?  Are the employees doing their tasks as they were expected to do? Are the tasks managers working efficiently?  Is the team working efficaciously?  Was the plan well conceived?  Are there new factors taking place that require modifications in the plan? Are the project deliverable leading to the expected project outputs? Are the project outputs leading to the expected project objectives? What are the outcomes and the impacts of the project? Is the project contributing as intended to the wider programme to which it was a component?  Are we learning from the experiences?

These judgment are possible as far as there is the possibility of comparing what is happening to what was expected to happen: therefore in order to carry out an evaluation we need at least two commonly visualized scenarios: the one that was expected (the program/project plan) and the one that is factual (the facts resulting from monitoring).   When comparison highlights difference between expectations and results, evaluation will attempt to find the causes of such discrepancies, attributing them either to lack of capacity of the implementing team, or lack of vision of planner, or to unexpected external factors, or other factors as it may be the case.  If the project is still running evaluations report will possibly indicate how to redesign the remaining actions in order to conciliate expectations and results. If the project is already completed, evaluations will indicate the  lessons learned from the project to be considered in the successive planning and implementation of similar projects. Evaluations are therefore an indispensable means for .programme cycle management and constitute an essential element of its actions.

While implementing a project the project team always need to carry on a certain level of monitoring and evaluation upon its own activities in order to constantly re-orient the work towards the expected results (see the section on project execution and control).  But self evaluation of the implementing team is not sufficient because development projects are part of a wider programme and therefore it is necessary that programme stakeholders understand how efficiently and efficaciously the resources devoted to the project have been used for the achievement of the expected project outcomes and impacts.  Evaluation therefore becomes a shared effort between various project stakeholders who want to understand how successful was a project and learn what are the factors that have to be considered in order to manage successfully follow up activities or similar projects in other contexts. While designing a new project and new programmes the lessons learnt from the evaluation of past actions is fundamental.  (See also  clarifying the relationship: Responsibilities of Project Managers and programme Managers).

In programme cycle management, evaluation is the last stage of each ending cycle and the first one of each new cycle.

 

The activities that project teams carry out must produce outputs that

will empower project beneficiaries to better interact with their

communities and utilise project deliverables to achieve the project

outcomes

These outcomes will enable the communities to tackle the problems

identified and produce changes in the factors generating these problems,

thereby contributing to the achievement of the project objectives

To make the project successful, it is necessary that project teams

understand the requirements, needs and problems of the beneficiaries

and that the beneficiaries understand the requirements, needs and

problems of the communities.

If the project team has correctly understood the needs and problems of

the target beneficiaries, they will be able to utilise the outputs to

empower beneficiaries, i.e. move from outputs to outcomes

If the project beneficiaries have correctly understood the needs and

problems of the communities, they will be able to empower communities,

i.e. move from outcomes to impact

Without active contribution of beneficiariers, project ouputs will not

be utilised to achieve project outcomes

Without active contribution of commmunities, beneficiariers will not be

able to utilise outcomes to achieve impact

 

An organization aiming at programme program quality should establish a system of Monitoring and Evaluation. In fact in order to ensure  program quality programme managers need to use the feed back of monitoring and evaluation in order to:

A good programme design will consider how to monitor and evaluate during and after the implementation phase (in itinere and ex post). It will therefore decide what are the indicators and what are the sources of information for judging the efficiency, the effectiveness and the impact.  Information sources may be independent from the programme/project management team and the organization. Others will be the internal evaluation tasks that will collect data concerning processes and outcomes.

Programme designers will consider the standard methods of collecting and analyzing  data and will adopt those that are seen as most suitable. Eventually a new approach may be proposed.   After deciding on the method there will be an illustration of the evaluation activity within the program/project, including the indications regarding how to use the monitoring/evaluation results for improved programme/project management for organization development  and for employee empowerment and learning.  (see  evaluation reporting methods; and methods for  generating acceptance for the evaluation reports).   The elements of an evaluation action are illustrated in the chapter on program evaluation.

 

Indicators

The purpose of evaluations is to empowering the different stakeholders to understand and learn all the project implications. Since evaluations are judgments, it is important that these judgments are based on documented facts and are interpreted at the light of values amongst project major stakeholders. If judgments happen to be perceived as arbitrary and subjective they cannot lead to meaningful evaluations.  While preparing a project it is therefore important to arrive at a consensus upon what will be measured during monitoring and evaluation phases, how it will be measured and against what parameters these measures will be interpreted.  What is going to be measured is called an "indicator".  A constellation of indicators quantifies qualitative processes and facilitate the description of the complex balance of social systems.  Development projects aim at improving the quality of life of the beneficiaries. And so the big question is "how to measure objectively and transparently the "quality of life"?  Only quantities can be measured objectively, judgment about qualities tent to be subjective to the value system of the judge. So in order to be "as objective as possible" we have to identify a constellation of quantitative parameters that can be actually measured and that have a logical relationship with the quality we intend to evaluate. Like in the case of the "health of a person" (that is a quality) we can use a number of objectively verifiable parameters that can help understanding the health balance (various physical analysis, psychological responses, etc.). In the same manner we can identify a series of economic, social and cultural parameters that can through light on the quality of life of the beneficiary population.

 

There are different kind of evaluations according if it is with reference to the effectiveness or the efficiency of the action, i.e. with reference of what qualitative difference has it made to the life of the beneficiaries and according to what qualitative standard has the action been carried out. (see later sections of this chapter). Accordingly in the logical framework we describe different indicators (and different sources of information) as they are chosen in order to evaluate how far the project was able to achieve the objectives (specific and overall), to produce the outputs (and their quality standard) and carry out the intended activities (with what level of efficiency).

 

In order to through light on project effectiveness and efficiency indicators should respond to certain "validity" criteria, i.e. they should be:

oxfam definition of indicators

 

----  Other resources from the web:

 

Some general definitions of evaluation:

 

Definitions of performance evaluation on the Web:

Definitions of process evaluation on the Web:

 

Definitions of outcome evaluation on the Web:

Definitions of evaluation criteria on the Web:

Definitions of program evaluation on the Web:

Definitions of program evaluation review technique on the Web:

  

 

In the course in Canada, they told us that for each specific objective (the level at which we measure outcomes), we must have a group of 3 - 4 indicators that can together measure the change we intend achieving for  that particular outcome. In order to measure the change (in behaviour, practices, standard of living, health, education, other impacts), data must be collected for each indicator at intervals (1 year, 6 months, etc.) and while evaluating, we assess what, where and how of baseline data and assess whether this change is being made

So, while establishing a monitoring system we must include collection of baseline data as one of the activities and decide who, what and where. During evaluation, we must assess whether this baseline data is really giving the data whereby we can measure the change we intend making. 

So, what do you think- we should have a chapter on monitoring and evaluation systems rather than one on monitoring and another one on evaluation? Even revising logical frameworks is part of the evaluation.

 

Definition by OECD  :    "Performance monitoring is a continuous process of collecting and analysing data to compare how well a project, programme or policy is being implemented against expected results.    Glossary of Key terms in Evaluation and Results Based Management, OECD, Paris, 2002

 

 

 

 

 

 

 

 

See also

Cost and Schedule Control

Motivating the Project Team

further reflections

Job profiles of an evaluation expert in a development organization

 

 

 

 

Definitions of Evaluation on the Web:

 

Definitions of outcome evaluation on the Web:
  • the systematic collection of information to assess the impact of a program, present conclusions about the merit or worth of a program, and make recommendations about future program direction or improvement.
    www.cdc.gov/tobacco/evaluation_manual/glossary.html

     

  • An evaluation used to identify the results of a program's/initiative's effort. It seeks to answer the question, "What difference did the program make?" It provides information about effects of a program after a specified period of operation. ...
    www.jrsa.org/jjec/resources/definitions.html

     

  • Outcome evaluations assess the reasons for differences between outcomes and objectives (eg, why the number and quality of permits issued exceeded or fell short of objectives). Outcome evaluations may include an examination of program processes and activities to understand how outcomes are achieved and how quality and productivity could be improved.
    www.epa.gov/evaluate/glossary/o-esd.htm

     

  • A comprehensive means of assessing the extent of change in the situation of a juvenile offender; a juvenile-in-need-of-services, and, where appropriate, their families; the effect of DJJ activities on the youth’s behaviors; and the cost of those changes. top
    www.justice4kids.org/glossary.htm

 

Definitions of impact evaluation on the Web:
  • A type of evaluation that focuses on the broad, longer-term impact or results, whether intended or unintended, of a programme or outcome. For example, an impact evaluation could show that a decrease in a community's overall infant mortality rate was the direct result of a programme designed to provide high quality pre- and post-natal care and deliveries assisted by trained health care professionals. See also "country-level impact assessment".
    stone.undp.org/undpweb/eo/evalnet/docstore3/yellowbook/glossary/glossary_f_I.htm

     

  • Impact evaluations assess program effectiveness in terms of end results (eg, Is the water swimmable? Why or why not?), including intended and unintended results. They may also assess the net effect of a program by comparing impacts with an estimate of what would have happened in the absence of the program.
    www.epa.gov/evaluate/glossary/i-esd.htm

     

  • A type of evaluation that determines the net causal effects of the program beyond its immediate results. Impact evaluation often involves a comparison of what appeared after the program with what would have appeared without the program.
    www.uwex.edu/ces/tobaccoeval/glossary.html