N —
\ & g = | 5 + / C. Va a i | | | j & Ba ‘ > | == 2 : g y | | . 7 ? '
tools for /
aoersity conservation
and t projects
r | Yorary ave ZOOM)
ar para i nye vty
Muse
SANBI Biodiversity Series 11
Monitoring and evaluation: tools for biodiversity conservation and development projects
The second in a series of project management handbooks, dealing with monitoring and evaluation as a pathway to learning in a people-centred development context
compiled by Cape Action for People and the Environment
Pretoria
2008
SANBI Biodiversity Series
The South African National Biodiversity Institute (SANBI) was established on 1 Sep- tember 2004 through the signing into force of the National Environmental Manage- ment: Biodiversity Act (NEMBA) No. 10 of 2004 by President Thabo Mbeki. The Act expands the mandate of the former National Botanical Institute to include respon- sibilities relating to the full diversity of South Africa’s fauna and flora, and builds on the internationally respected programmes in conservation, research, education and visitor services developed by the National Botanical Institute and its predecessors over the past century.
The vision of SANBI: Biodiversity richness for all South Africans.
SANBI’s mission is to champion the exploration, conservation, sustainable use, appreciation and enjoyment of South Africa’s exceptionally rich biodiversity for all people.
SANBI Biodiversity Series publishes occasional reports on projects, technologies, workshops, symposia and other activities initiated by or executed in partnership with SANBI.
Technical editor: Emsie du Plessis Design & layout: Daleen Maree Cover design: SANBI Graphics
————
| How to cite this publication CAPE ACTION FOR PEOPLE AND THE ENVIRONMENT 2008. Monitoring and
evaluation: tools for biodiversity conservation and development projects. SANBI Biodiversity Series 11. South African National Biodiversity Institute, Pretoria.
ISBN 978-1-919976-47-1
© Published by: South African National Biodiversity Institute
Obtainable from: SANBI Bookshop, Private Bag X101, Pretoria, 0001 South Africa. Tel.: +27 12 843-5000. E-mail: bookshop@sanbi.org. Website: www.sanbi.org. Printed by: Creda Communications (Pty) Ltd, PO. Box 9403, Johannesburg, 2000.
FINE GCC SPecreneecceccecrter sates ine cry Minton ernie treet = cer MEST cnr are eae ener eee ener ee Vv LOSSES i ecereaet ncreestoapen Be pet acne iin tots So Cone Ae | A IO Be RRO rein A: Res St Ae oe enn a Vv NC HIMMEI GP CTINE TICS amraertsce aren ter caper scree Soe RON ate cea y secs esa casa veussaouamerotes oat ee iss nanan tee ssas sass bavepuasowel oantinsaa¥ilsa astiaccuu genet vi JAE ROUT OS cepenatbeh wrcecetcen ase Teeter Coane cer eEL TOOT RECT eee ee ee ee Vii JEU, - cette DS ele SS eT) AN (8 cae oo tere incor eee, en Eo ea eee ee een Pet ae vii INTRODUCTION 1 be MAAIDICTERRLIIS [VAS] [2 10S 0 a OS Re ep ae Re ere ae Sn eer |
2 ages Sot a eg TCG IS gM bl bod ig See acres Oe ie Se ee EE Core Won nck ce Be he Re 2
Db) MAE within tiie’ DIOGIVensity COMSEFVAtlON SECON .c......c.sacenesctscseecesvoreessecsaceceanservecrdvocsersecuscnsecsenssntsetecescavoesediees 3 2 MAW KOUIISSULE IS TVAINIBIENOYO) G13 OT; U0e siteanete pepe tec nse Romar Te aun ey SOP Permernnoh tenon tea ere ere ne err eee 6 SON VaLOULEAN WSEs IS FUAIND BOOKS sits casccctescnscateseccuens exces duces sos sospotensesseptcsictanece usberd auvasuuactshusodi beidaraseardsvees 7 A LOVES TAN D BOOK IS. SiiNUI GM WRED ctecccsosscenescsceccotzssessastacteserorstrscscsczsessereasesestcocsotensodsosoaiestocadussusisestedeeoes 8 Section 1: DEVELOPING AN Maé PLAN 11 imei yak cide: OBJECTIVES @RMREIPROJ ECM 2 | sccsceccecscscccoscevccxrcccsastcesecosecercta seas eecsteoeestiscuteateseeseesuevencsecneoreeecers 12 Pee BOWIE ESE Mite INS ORMAIION COEEECTIED?. si. ccucccecccceccscasccesceccsseetnceerecceecasee ase ceaeetesaseeseteaecheaesecnoare 17 Se HOV ARE THE INDICATORS USED AND! DEVELOPED? ciiicet-cicxcescracccccecesstexececnousnscdscsansecsuscecseoscseasdesasetesnscucoes 20 PVA Mend ODS VIEL BE WUSED TO (GATHER INFORMATION? vacecscccscccoccscecsecesesascaccecceossassssesssnseascosnsoestzcees 28 os) WAM ROUVAYITETEST EAN ROCCE E7610 STN ln 6 Sa eaten oe aA RP BN rt EN o-oo eRo e 30 WACOM al INAV Vile eagtysn ic MIAN Ge tN etna cote crcecenn vec cosecseccectesscieessSostsccustschcosasuccotcueeseas neta oaocre sate tanec ths nouseseesaaseeceaae neneoeede 33 TeV ONY Medes 1c: Si SE UB IAIN ANGE ccs anc csnecscccvncctectcesenscus nats anera sis anatnssasdhcteosisnusnstessicessxceceensazeueedee 35 Pes nOVy Wile tEARNING BE BUIET INTO THE M&E/PROCESS? 2228.8 cts see cccscs. access ansansteectess vas scasssayeeseustencceoxe oy, Section 2: LEARNING 40 Pel OVy DO ESMEARINIIN Gr IARBEND Herc. cs.tesscstschdaccsecs tcasesstonsstestssesessotsdsetensiees avebeodaastimavicorctntescreaitsalcoeaduveste Mdesnsemeuies 4| DP aUEARININGHIN EEE PROJECT EAM ses cos sscccsucvesceseetcne-nsccentancesutee-stsaseves eceeseusectbstcnereesetrtctreecucoaccosccnvosrnasttitntaxecess 43 PS eB UIEDINGA LEARNING IENVIROINMENT, | scsszcccqscecctiectecsscscacvecsfecckehccsctecnstusnausctcctnes-dcetnsbucccocccasnsbasasrasuvancevactuccheve 50 Section 3: GATHERING THE INFORMATION 02 Bal eONDERSTANDING YOURCINFORMATION NEEDS” ws icscsccesacecteastescassacueneatossacsevestuzcscntscoatuensrtcatotoutestloteckoccotessueadees 53 Be SELECIINGAND USINGAPPROPRIATIE METHODS 5 sccasce-csoses toetnet <cvceraenseneresnascevaestevdenventrossdtseasticsenadtecvcestectens 57 3.3) MANAGING THE INFORMATION GATHERING SYSTEM i.c...c.cctccccteccaccestcctsencesecercecerevascosescoccstacascatorcorsesesaccceesas 63 Section 4: ANALYSING THE INFORMATION 67 An PESTABLISHING IE THEREARE SIGNIFIGANTT DIFFERENCES 1.....ccccccsccscccsnsecscecssocrercecessientcustesesescuovensensarssiasansass 68 4.2 CLARIFYING AND AGREEING ON THE REASONS FOR ANY DIFFERENCES. o.....esescesessssesteseseseeeneeteneenenes 74 Aa CONDUGTINGIFORMAL EVALUATIONS. © caccsciesessvsnssacsnvscssvensczustesneasversnonevanertrauccsloybaveangoxsbneteatsssbessasGoustceetseneabenani 76 Section 5: ACTING ON THE ANALYSIS 83 Ea DAVY INGO Oil RECOMIMENDATIONS 2 ccsccccscvescrasescovevcusoxssxoss'eqnavtenscusoxvextdrtstunsenhcsrsucarscoseveutestuasiveccveveudavosessescucats 84 Epa ME EE Oa HL Fat SISTINE ANISIIN Gi aces cco cos vepagis se cc c05 casns ca converasucestecdesconesenved Coney na shostexcuechussverdasedysanunanstiskaasakanausaccers 87
Section 6: SHARING THE LEARNING (REPORTING) 93
Bl WAT IS ACCOUNTABILITY? . cccssccvsccessssssescccssosnsusscacnsonsusubessrsstssssneonavsnnrasazensaneansiapsnedses is6:ssi0i/0skteGntceranaeenen GEESE 94 BD REPORTING FORMATS | siccsccsccssocsersercescosronscccnsonavazncucoctpacartecvanesobunstcarenssasenscensercentstenarisonetespensistes traseeS ieee nea nEnEeene enon 99 | 63 FINDING COMMON GROUND: ssccssecccsossiersessssesciscspocosescsntossencanceonessoraseseesossronsorenetebesteneevsnenconeneeteneeneeten a neon ENRORStE 102 ) CONCLUSION 107 FURTHER READING 109 BIBLIOGRAPHY 11 APPENDIX 1: GUIDELINES FOR TRAINERS 113 ; INTRODUCTION: ciciecicsecnsciccsle ee hlpeacetecsciiaieiesetlicscs BLT een ne 113 Section’ 1: DEVELOPING AN M&E PLAN c..ccccccccsoscossccnssceccovccccaszccesccescosececsversesStieucosssetns nee one 114 Saction 22 LEARNING. .ccccsscéscsacccssccsssonsocsnvenvoxscsssateooncossascasasccucensoseenstersesesosanhecae caps eieseeoas Grete Nie TT Tone I AI7/ Section 3: GATHERING THE INFORMATION. scsccssecccssseccscssscossesecsdssesdessapseecereeste os eelrint eee 120 Section 4: ANALYSING THE INFORMATION .u.ccssssosccsssseccosscssscezesesesessosesssvosetsaestete tee renaey ee ee 123 Section 5:ACTING‘ON THE ANALYSIS): ncccscsicscccscecscconearerocsttceszcuceveececoueets catch ieee eee 125 Section 6: SHARING THE LEARNING csscccccssssscccccecvececcccovesssstacccacnccsscsvgrccusstavses tstoesinv setae 126 eee
We are pleased to present the second in a series of three handbooks that form part of the C.A.PE. Part- ners Toolbox, following the first publication, Project planning: tools for biodiversity conservation and development projects, published last year.
We hope that this second publication, focusing on project-level monitoring and evaluation (M&E) and re- porting, will be of use to organisations and projects across the C.A.PE. partnership.Although some projects have M&E systems in place, there is always room for improvement in tackling the big questions around the real impact of our biodiversity conservation work—in terms of both the state of biodiversity and the deliv- ery of socio-economic benefits.
This resource sets out practical tips for developing an M&E plan, building a learning environment, setting up an information gathering system, analysing the information, drawing out recommendations and sharing the learning. | believe it will be an asset to anyone who is designing, implementing or improving an M&E system. Guidelines for trainers are also provided to enable you to use the material in the book further within your organisation.
| wish you the best of luck in using this resource and hope it will contribute to the development of excel- lence in monitoring and evaluation throughout the C.A.PE. programme.
Dipolelo Elford Chairperson: C.A.P.E. Implementation Committee June 2008
The Critical Ecosystem Partnership Fund has supported the production of this handbook as a resource that adds value to our investment in the Cape Floristic Region biodiversity hotspot and to the C.A.PE. programme.
CEPF believes that it is critical for all of us in the conservation sector to analyse the overall impact of our work in making progress towards our intended conservation outcomes. This involves setting clear and coherent objectives and strategies that give our projects focus and direction, an expected results chain or logical framework, measurable indicators that meet our monitoring and reporting needs, and systematic and regular processes for collecting and managing data.
All these topics are covered in this handbook, using real case studies that bring the theory of monitoring and evaluation to life and make it accessible. We welcome the addition of this handbook to the C.A.PE. Partners Toolbox and encourage readers to apply it to their own projects and programmes.
Nina Marshall
Grant Director
Critical Ecosystem Partnership Fund June 2008
Acknowledgements
This handbook is an adaptation of the Planning for monitoring and evaluation handbook by Michael Randel, published by Olive Publications in 2002. Cape Action for People and the Environment (C.A.P.E.) obtained permission for the adaptation, and commissioned this handbook in 2007-2008.
lt is preceded by and builds on the material contained in the Project planning: tools for biodiversity conservation and development projects handbook prepared by C.A.P.E. and published by the South African National Biodiversity Institute in 2007.
In the course of developing this handbook, a number of people made valuable contributions. In the early stages, the User Reference Group (comprising Tanya Goldman and Onno Huyser) and the Client Reference Group (comprising Mandy Barnett, Azisa Parker, Caroline Petersen and Monique Damons) provided a clear foundation for the development of the handbook.
Project managers from three projects and a small grants funding agency generously provided time and insight for the purpose of generating case study material. They are:
* Cape Flats Nature.
* The C.A.PE. Conservation Stewardship Programme, located in CapeNature.
* Greater Cederberg Biodiversity Corridor.
* Table Mountain Fund.
Detailed information about these projects can be found in Fynbos fynmense: people making biodiversity work (Ashwell et al. 2006, SANBI Biodiversity Series 4, published by the South African National Biodiversity Institute, Pretoria) or on the C.A.PE. website, http://www.capeaction.org.za.
Participants in the Project Developers’ Forum held on 18 and 19 March 2008 also contributed material used in the examples in the handbook and in further refining the content.
The C.A.P.E. Co-ordination Unit oversaw the process of producing the material for the handbook. Material was developed and compiled by Carol-Ann Foulis and Jenny Whitehead. Sue Soal skilfully facilitated the Project Developers’ Forum and helped with the conceptualisation of the handbook.Anne Kroon brought her creativity and experience to bear in helping to develop the Trainer Guidelines. Judy Norton meticulously edited the handbook. Michael Randel, as the original author of the Planning for monitoring and evaluation handbook, provided in-depth and valuable comments on a draft version of this handbook.
SANBI’s Publication Unit was responsible for technical editing, design and layout, and cover design for the handbook.
Financial support was provided by the Critical Ecosystems Partnership Fund and the Global Environment Facility through the C.A.PE. Biodiversity Conservation and Sustainable Development project.
The C.A.P.E. Programme is hosted by the South African National Biodiversity Institute and is supported by 24 signatory partners.
vi
AAR After Action Review
C.A.P.E. Cape Action for People and the Environment CDRA Community Development Resource Association CEPF Critical Ecosystems Partnership Fund
CFN Cape Flats Nature
CFR Cape Floristic Region
CNC Cape Nature Conservation
EIA Environmental Impact Assessment
GIS Geographic Information System
LFA Logical Framework Approach
M&E Monitoring and Evaluation
NGO Non-government Organisation
OvI Objectively Verifiable Indicator
PCM Project Cycle Management
PLA Participatory Learning and Action
PME Participatory Monitoring and Evaluation PRA Participatory Rural Appraisal
SA South Africa
SANBI South African National Biodiversity Institute SCM Sustainable Conservation Management TMF Table Mountain Fund
TOR Terms Of Reference
WWE World Wide Fund for Nature
Adaptive management Management that is responsive to learning.
Action learning A learning process that, by reflecting on past experiences, draws out relevant lessons and supports the preparation for future actions.
Activities The steps that the project must take to achieve the Outputs.
Assumptions External factors, not under control of the project, that are identified as being necessary for project success; part of the project plan (LFA).
Beneficiaries The people who will experience improved conditions (benefits) as a consequence of the project targeting their needs.
Benefits The positive conditions of change resulting from a project. Development goal Describes the benefits that will result from the project. Development objective See Development goal.
Enabling factors See Assumptions.
Ex-ante evaluation An assessment of the project plan and approach, before it is implemented, to assess whether it has a good chance of achieving its objectives.
External factors See Assumptions.
Vil
Formative evaluation A periodic review of the project that allows continuous feedback to inform ongoing changes and improvements in a service or a product.
Immediate objective See Project purpose. Impact The positive differences resulting from the project; often seen in the benefits for specific groups.
Impact evaluation An external and in-depth study of the impact of a project on its beneficiaries; usually carried out 3—6 years after the project has closed.
Indicator The measure that is used to assess if an objective has been achieved, or what progress has been made.
Inputs The human, financial and material resources required to implement the project. Intervention A deliberate action on the part of the project to influence change in a social system.
Logical Framework Approach (LFA) A method for project planning that focuses on objectives or outcomes.
Means of verification The location or source of the evidence used as indicators of project achievements.
Mid-term evaluation An external assessment of the project, usually conducted midway through the life of the project, that focuses on its performance and assesses whether the objectives continue to be relevant.
Objectives Describe what we are trying to achieve. There are four levels of objectives in the LFA approach: Goal, Purpose, Outputs and Activities.
Objectives analysis A tool to visualise an improved future, linked on a,‘means-end’ basis.
Outcomes The positive changes that result from the project’s intervention; achieved by the use of the outputs of the project.
Outputs Describe the responsibilities of the project, and the services and products it will deliver.
Participatory Monitoring and Evaluation (PME) An approach to monitoring and evaluation that emphasises the role and views of the target groups and beneficiaries of the project.
Project A set of planned activities designed to achieve specific objectives with given resources within a specific time frame.
Project Cycle Management (PCM) A process of managing a project from the start of the preparation and planning phase, through to implementation and evaluation.
Project purpose Describes what the target group will be able to do because of the support they receive from the project.
Results See Outputs.
Summative evaluation Evaluation that takes place to review the accomplishments of the project and to inform high-level decisions of future funding and policy.
Sustainability The ability of something to maintain its value over a long period, without external support.
Target group The people whom the project aims to support, and whose consequent actions will bring about a benefit to themselves and/or others.
Terminal evaluation An evaluation at the close of the project on whether it has accomplished its objectives or not.
Vili
;
Monitoring and evaluation: tools for biodiversity conservation and development projects I
INTRODUCTION
1. WHY THIS HANDBOOK?
Most people working in the biodiversity conservation sector have their hands full with tasks and deadlines associated with project implementation. This handbook focuses on a particular set of activities that are integral to the process of implementation, but that are also distinct from it— namely the activities that make up monitoring and evaluation (M&E). These activities have a special quality—they are about observing and gathering data, about reflecting and learning. Through this process, M&E has an important contribution to make to the success of projects. It can help to cre- ate ones that are innovative and pioneering, and impart a different way of thinking, doing and relat- ing.Another way of describing M&E is that it is ‘learning while doing’.You do not stop implementa- tion to ‘do’ M&E. It is part of the process of implementation.
This handbook is the second in the C.A.P.E. Partners Toolbox series. The first handbook focused on project planning and drew on the Logical Framework Approach (LFA) as the basis for managing the planning and implementation of projects. LFA terms and concepts form the foundation for this handbook on M&E. Together, these handbooks are part of building a shared language about project management within C.A.P.E.We hope this will enable project partners to talk to one another, that it will fuel discussion and debate, raise questions as well as provide useful tools and material. There is no one way of doing M&E (or planning). The intention is to help you become more confident and familiar with M&E so that you can adapt and change the tools and methods to suit your context.
The focus of this handbook is on project monitoring and evaluation and what is associated with it: clear objectives, a particular form of support or intervention, a set time frame, a defined target group and beneficiaries.
The notion of target group is central to projects—who is it that the project aims to support? What are the new actions that are required to bring about a benefit to the environment, to themselves and/or to others? Beneficiaries are also important—who or what stands to benefit from these changes? Within the context of biodiversity conservation, both the environment (or biodiversity) and people can be viewed as beneficiaries.
The C.A.PE. programme is characterised by a diversity of projects at different stages of organi- sational development and project implementation. The scale and complexity of these projects is wide ranging, as are the resources that are available to them.What is common across the projects is a growing interest in developing M&E capacity and building an M&E ‘practice’ that is identifiable within the C.A.P.E. programme.
2 Monitoring and evaluation: tools for biodiversity conservation and development projects
TERMINOLOGY/AT/A\GLANCE: PRACTICE
There are a number of characteristics of a practice. It comprises a set of activities in which people engage on a regular basis, and the behaviours that accompany these activities. More specifically, it is work (or activities) that takes place within a particular discipline with its own body of knowledge, ways of doing things and code of conduct that have been developed and tested over time, e.g. conservation practice. For this reason, it has the power to hold its practitioners to account. However, to maintain the vitality of a practice, practitioners are constantly contributing to it and changing it along the way. Lastly, a practice (in whatever field) is recognisable to others working in the same field.
Thus the notion of ‘practice’ can be applied to the field of M&E, and practitioners working within it.
(Collingwood 2007)
This handbook aims to provide you with:
* An overview of some of the key issues in project M&E. * A guiding framework within which you can develop your M&E plans.
r i i 1 1 1 1 ' " Tools, concepts and exercises to build your own M&E practice. 1
=
The handbook will focus largely on the monitoring and evaluation activities carried out by project staff.
a) Recent trends in Mae
The world of M&E has become increasingly specialised terrain, partly to address the complexity and sophistication of development projects more adequately. This can sometimes have the effect of limiting who can and cannot do M&E. There are, however, some new and exciting trends that firmly locate M&E in the hands of those who are responsible for project management and implementa- tion. We will look at two trends.
The first is a growing recognition amongst the various development players that a key aim of M&E processes is that of learning. If people, projects and organisations do not learn from their involve- ment in and contribution to M&E activities, then it has failed in one of its core objectives.
The effect of this has been to bring learning to the fore in newer models of M&E, by placing greater emphasis on the importance of reflecting on the lessons of implementation and learning from them. This indicates a shift away from upward accountability as the driving force in the rationale for project M&E.‘Accountability’ remains a key issue. However, this concept has been extended to that of multiple accountability, focusing on horizontal and downward accountability as well.
A recent nine-month study, commissioned by the Ministry of Foreign Affairs of the Netherlands, looked at the issue of M&E in its theme-based co-financing programme. It had this to say about where M&E has come from:
Monitoring and evaluation: tools for biodiversity conservation and development projects 3
‘Historically, accountability orientated M&E has often focused primarily on upward accountability for the expenditure of funds and the delivery of outputs.Attention to outcomes and impacts and how feedback
mechanisms could be used to help those implementing programmes improve their performance has been minimal. (De Ruijter et al. 2006: 9)
Many donors and development organisations are now grappling with how to re-think M&E to make it more relevant, and to increase its contribution to the impact of development projects, and to the sustainability of these impacts.
The second positive trend is to view monitoring and evaluation as two sides of the same coin—‘as an integrated process of continual gathering and assessing information to make judgements about progress towards particular goals and objectives, as well as to identify unintended positive or nega- tive consequences of action’ (De Ruijter et al. 2006: 10). In this scenario, project managers and im- plementers actively value their work and assess its contribution to development goals and project outcomes. The alternative is to lose the meaning of their work:
“The separation of monitoring from evaluation has been partly driven by the classical approach to develop- ment projects, in which evaluation was undertaken every now and then by external experts, while monitor- ing was the task of project implementers. It is exactly this scenario that has resulted in an inability of many development initiatives to learn effectively as it disconnects the information collection from the sense-mak- ing that precedes improved action’ (De Ruijter et al. 2006: 10).
Even the separation of ‘planning’ from ‘M&E’ can set up an unhelpful distinction between two proc- esses which, in practice, are intimately located and held together under the umbrella of project management.
These shifts within the field of M&E challenge us, as development practitioners, to think more deeply about our practice and to find ways to articulate and describe this to others. This means having a clear ‘approach’ to the work we do—a way of describing what we do and how we do it, based on reality (Soal 2001).Wéithout this frame of reference, it becomes difficult to talk about development work and its impact both amongst ourselves (people working within projects) and to donors. Two questions become important:
‘Do we know what is valuable about our work?’ and ‘Do we know how to go about valuing our work?’
This means that any M&E framework has to begin by asking, ‘What is our work?’ and more specifi- cally, ‘What is our practice?’. This requires an openness to one of the building blocks of M&E proc- esses and systems: reflection. We have to reflect in some way or form, individually or collectively, formally or informally, on what we are doing.
Much of the recent literature on evaluation therefore refers to ‘practice improvement’ and the role that evaluation can play in contributing to it.
b) Mae within the biodiversity conservation sector
The task of conserving the rich biodiversity of the Cape Floristic Region (CFR) is a large and sig- nificant one, as is C.A.PE’s contribution to it.
4 Monitoring and evaluation: tools for biodiversity conservation and development projects
Working in this field, we must be able to demonstrate progress—go beyond implementation to show results and effects. This requires understanding and working in an increasingly complex ter- rain that combines ecological, social, political and economic factors.
In M&E in biodiversity conservation, many of the objectives are quantifiable. Perhaps this is not good be- cause it can mean that we do not spend enough time thinking. Maybe we have to look at this?
(Conservation Partnerships Programme)
Researchers looking at the usefulness of M&E within the conservation sector argue that (1) good project management is closely linked to effective M&E systems and (2) good M&E contributes to
better decision-making and therefore enhances the success of biodiversity conservation (Stem et al. 2005).
Stem et al. (2005) propose four (sometimes overlapping) reasons for undertaking M&E within the conservation sector:
* For basic research purposes—to gather or generate knowledge about a subject to understand it better.
= For accounting and certification purposes—to assess the extent to which a programme or project is fulfilling its obligations to donors, the public, government or other enforcement enti- ties.
= For status assessment—to assess the condition or status of a particular conservation entity (species, population, ecosystem) at a particular moment. This is generally irrespective of a spe- cific intervention designed to affect the variable. Tools associated with status assessment include population monitoring, rapid assessments, state-of-the-environment monitoring, and report cards and scorecards. This kind of monitoring is often indicator driven.
= For effectiveness measurement—to measure effectiveness of discrete interventions employed by specific actors. These can be divided into two categories: impact assessment and adaptive management.
This handbook tackles issues of M&E that largely fall under ‘effectiveness measurement’. In draw- ing on the LFA and project approach to M&E, the handbook links with the notion of ‘interventions employed by specific actors’ (Stem et al. 2005). You may also find that some of the principles in this handbook dovetail with those of ‘adaptive management’.
TERMINOLOGY/AT/AGLANCE: ADAPTIVE MANAGEMENT;
According to Wikipedia, adaptive management originated in the 1970s out of the work of two ecologists, Holling and Walters, at the University of British Colombia, Canada. It is defined in the following way:
Adaptive Management (AM), also known as Adaptive Resource Management (ARM), is a structured, iterative process of optimal decision-making in the face of uncertainty, aiming to reduce uncertainty over time via system monitoring. In this way, decision-making simultaneously max- imises one or more resource objectives and, either passively or actively, accrues information needed to improve future management.AM is often characterised as ‘learning by doing’.
Monitoring and evaluation: tools for biodiversity conservation and development projects 5
There are a number of features associated with this approach:
= Iterative decision-making (evaluating results and adjusting actions on the basis of what has been
learned). = Feedback between monitoring and decisions (learning). = Embracing risk and uncertainty as a way of building understanding.
Using Bayesian inference—formulating a hypothesis and then collecting evidence that supports or refutes it.
(http://en.wikipedia.org/wiki/Adaptive_management)
= Adaptive management is like...dancing on the back of a truck while it is moving. You have to be fit and agile! \G
(Project Developers’ Forum, |8—19 March 2008)
6 Monitoring and evaluation: tools for biodiversity conservation and development projects
2. WHO IS THIS HANDBOOK FOR?
There are two audiences in particular. e Project developers, managers and implementers
These are people who are leading and managing the development and implementation of projects and are responsible for thinking about how to do M&E for their particular project. Most often, they are called the ‘project manager’ but they may also be project staff who are leading a particular aspect of the project.
Because the handbook presents an overview of an M&E system, it is useful for project managers and project staff who are new to their positions and are looking for ways of ‘getting started’. There are also sections in the handbook that are more challenging and that will extend project managers and project staff who have some experience of M&E.
What characterises this audience is that members are self-motivated to learn and to develop their skills and understanding of M&E.
e Intermediaries
These are people who play a supporting role to projects and project staff. They can broadly be categorised as ‘capacity builders’. What characterises this audience is the role they play in supporting the learning of others. In some cases, they may be the project manager who is located within the project. In other cases, they may be external to the project, such as the project developer.
Their purpose in using this handbook would be to develop and deepen their own understanding and skills in M&E so that they can build the capacity of their colleagues. The design of the handbook takes this purpose into account as well.
We have made certain assumptions about you as a reader:
* You have prior working experience and some experience in biodiversity conservation projects.
= You have tertiary-level qualifications and are comfortable engaging with written texts. * You occupy positions where you are willing and able to apply the learnings and have the
opportunity to do so.
We have also assumed that the audience comprises a mix of scientists and social scientists, and that part of the purpose of this handbook is to build a common language for M&E.
Monitoring and evaluation: tools for biodiversity conservation and development projects 7
3. HOW YOU CAN USE THIS HANDBOOK
You can use this handbook in a number of different ways.
e Proactive learning
You might be reading this handbook, alone or as part of a reading group with your peers, to learn more about project monitoring and evaluation.
e In response to a problem
You may recently have had a problem in your work, and are concerned that your existing M&E processes failed to give you advance warning of this problem. You might be looking for ways to improve your project’s ability to monitor and learn from experience.
e In response to an opportunity
You may be in the midst of preparing a new project, and are considering how to include M&E concepts and tools into the work. You may be interested in practical ideas that can be part of your approach from the beginning.
Depending on your need and interests, you may choose to use this handbook in a number of different ways. You may have the time to read it from cover to cover, or you may dip into it from time to time, finding ideas that will help you deal with your particular questions. We would encourage you to read it in the sequence in which it is presented.
This handbook can also be used as a resource by your organisation or project team. For example, parts can stimulate discussions on your approach to project monitoring and evaluation.
Monitoring and evaluation: tools for biodiversity conservation and development projects
4. HOW THE HANDBOOK IS STRUCTURED
The Monitoring and evaluation: a pathway to learning handbook will take you through six steps that outline an approach to project monitoring and evaluation. We will present the objectives for each section, some examples from the field, relevant ideas and tools to stimulate your thinking. There will also be exercises for you to complete.At the end of each section, we provide a list of References and further reading.
&
Developing an M&E plan
Thinking through the demands that are likely to be made on your M&E system is the first step. Through thinking about what kind of system you need, you can develop a framework that brings it into being. You will explore a number of important issues that will influence your approach to project M&E.
Learning
Learning forms the background within which M&E happens. By understanding some of the key concepts and processes associated with learning, you can start to create the environment that will encourage and facilitate learning.
Gathering the information
Information is the fuel that drives the M&E system. You will have to collect information on an ongoing basis. The indicators for your work will tell you what to look for. There are a large number of methods you can use to collect both quantitative and qualitative information. Depending on your information needs, you might be looking for primary information (new information that you are the first to collect) or secondary information (information already collected by other groups or organisations that is nevertheless useful to you).
Analysing the information
You will have to make sense of the information that you have collected. This means exploring what happened and why it happened as it did, looking for reasons that helped or blocked the achievement of your objectives.At this stage, you could conduct a formal evaluation to deepen this analysis further.
Acting on the analysis
The whole M&E process has been building towards this step. By drawing on the information
you have collected, and the insights you have developed, you can now decide what follow-up action should take place. This often takes the form of recommendations. It also requires effective decision-making that is supported by the various stakeholders.
Sharing the learning
Lastly, as part of your accountability, you will have to document what you have noticed and plan on changing, and communicate these findings to project stakeholders. If you have learnt something significant from your experiences, you may want to think about how you can share this with interested people in your own and other organisations. Reporting to funders is also a key activity in this step.
Monitoring and evaluation: tools for biodiversity conservation and development projects 9
Things to look out for
To illustrate the application of various ideas and tools from the field of M&E, we have used examples from three C.A.PE. projects throughout the handbook. We have described these projects in generic terms, i.e. Urban
Sustainable Conservation Management Project, Conservation Partnerships Programme, Biodiversity Corridors Project and Small Grants Funder.
Resources
This part of the handbook provides you with tools and ideas to help you with the practical ‘how-to’ aspects of project M&E. Some of this material is presented as categories, concepts and guidelines.
Exercises
We have used exercises to encourage you to reflect on your current thinking and practice in relation to
M&E. These exercises provide you with an opportunity to make notes about any new ideas or thoughts you have about M&E and how you apply them in your project.
REFERENCES AND FURTHER READING
COLLINGWOOD, C. 2007. Revealing practice, re-imagining purpose, claiming our place. Biennial Practice Conference: McGregor, May 2007. http://www.biennial.cdra.org.za (accessed January 2008).
DE RUITJER,A., DIETZ,T..VAN GONGEN, E., HELMSING, B. & KNORRINGA, P. 2006. Evaluation of the Theme-based Co-financing Programme. Cross-cutting study: monitoring and evaluation. Prepared for Ministry of Foreign Affairs of the Netherlands. http://www.tmf-evaluatie.nl/eindrapportage%20M&E.doc (accessed January 2008).
SOAL, S. 2001. How do we know what difference we are making? Reflections on measuring development in South Africa. Adapted from a presentation to a workshop between Christian Aid and South African partners, Durban, October 2001. http://www.cdra.org.za (accessed January 2008).
STEM, C., MARGOLUIS, R., SALAFSKY, N. & BROWN, M. 2005. Monitoring and evaluation in conservation: a review of trends and approaches in conservation biology. Blackwell, Gainesville, Florida.
Monitoring and evaluation: tools for biodiversity conservation and development projects 11
Section 1: DEVELOPING AN Mae PLAN
The M&E plan provides you with an overall framework for your M&E activities. Its purpose is to remind you of what you are trying to achieve through the project, and help you to think through your information needs to effectively monitor, evaluate and manage your project.
The purpose of planning is the process, not the artefact.
(Small Grants Funder, citing Eisenhower, 1890-1969)
We have identified eight key questions that will help lay the foundation for a good M&E plan: |. What are the objectives of the project?
Who will use the information collected?
How are indicators used and developed?
What methods will be used to gather information?
Who will participate in M&E?
When will M&E take place?
How will the M&E system be managed?
i ee er)
How will learning be built into the process of M&E?
, By the end of this section, you will have:
" Explored eight questions that will assist you in developing or revising your own M&E plan.
Been reminded of the links between M&E and project planning.
Reflected on what you are currently doing and why you are doing it.
A better understanding of the role of objectives and indicators in project M&E.
; 1 1 1 i 1 1 1 1 1 1 1 Deepened your skills and understanding of how to develop indicators. , 1 al
The devil's in the detail
It took us a couple of months to develop our logframe.We had an extensive participatory process, which meant we wrote and re-wrote the logframe. Because our project was about mainstreaming biodiversity, there were a variety of voices that we had to listen to. It was furthermore a high-risk project, which made the funder nervous. This meant that we had to be more specific. We sometimes felt as if we were in a ‘catch 22’ situation with the detail. Initially, | was holding people very tightly to the logframe. But in the end, it allowed us to achieve huge
things very quickly.
(Urban Sustainable Conservation Management Project)
ZS
«
SZ
«
12 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.1 WHAT ARE THE OBJECTIVES OF THE PROJECT?
An important starting place for any M&E plan are the goals and objectives of the project. If they are clear and specific, then you can be clear about what you are trying to monitor and evaluate.
Behind this simple statement lies the experience of trying to monitor and evaluate projects that do not have clear objectives for the work they are doing. In such cases, it is difficult to know what information to collect.As a consequence, it is almost impossible to assess whether or not the project has succeeded.
While some projects know what they want to do, they are not always clear about what they will achieve through this work. For this reason, planning methods like the Logical Framework Approach (LFA) and Results-based Management have been developed. These methods place a great deal of emphasis on first clarifying the desired destination (the Goal and Purpose) before you work out how you will reach it (through the Activities and Outputs).
This requirement of clear plans should not stop you from experimentation and piloting. It is still possible to try out new ways of working and to have a plan that is clear about this.
Different types of objectives
The Logical Framework Approach (LFA) method (as introduced in the C.A.PE. Project planning handbook) outlines four main levels of objectives. They are linked in a hierarchy. These different lev- els are important for project M&E as they help distinguish between a project’s outputs (the work carried out by staff) and its outcomes (the changes that result from the project).
Often more time is spent on monitoring outputs because it is within the control of the project and therefore easier to measure on an ongoing basis.As a project manager, you are able to see what goods, services and products the project is making available to the target group. It is much harder to measure what the target group is doing with your goods, services and products, or to measure the longer-term impact it has on the broader group of beneficiaries, or on the environment. Yet it is important to the success of a project to assess the outcomes, as it is the only way to know wheth- er all your work (the outputs) is effective.
TERMINOLOGY/AT/A(GLANCE: OUTCOMES
The Development Goal and Project Purpose are often jointly referred to as Outcomes This is the approach that will be used in this handbook.
Monitoring and evaluation: tools for biodiversity conservation and development projects 13
Here is a simple description of these four levels of objectives:
This describes the longer-term benefits to which the project will contribute, either through a change in the environment and/or through its contribution to Development the lives of beneficiaries. The Development Goal normally has a clear link to the Goal initial problems that the project was set up to address. There is normally only one Development Goal in the project plan. Other projects may be making a different contribution to the same Development Goal.
This expresses the actions that the target group will take to bring about the desired change (identified in the Development Goal). The Project Purpose
Project Purpose often describes a change in the behaviour of the target group, or changes in the situation in which they are located. It is unusual to have more than two (or at most three) Project Purposes in the plan.
These are the goods, services and products that the project makes available
Outputs to the target group.A manageable project will normally have between five and eight Outputs. The various tasks that the project staff carry out to deliver the Outputs to the Activities target group are called the Activities. Each Activity is clearly linked to a specific Output.
The term ‘target group’ is used here to indicate the specific groups and organisations that the project is trying to support. This recognises that a project often cannot assist everyone directly, but can work with and through others (the target group) to help initiate and support change in a wider group (the beneficiaries). Within the context of biodiversity, the notion of ‘target group’ is significant in bringing to the fore the attitudes, values and behaviours of people that have to change in order to achieve the objectives of biodiversity conservation.
Project M&E is also interested in two additional areas that are often found in project plans.
These are the resources that are needed to implement the project. Inputs will
Inputs include financial and human resources, and physical resources such as land. They may also include contributions from projects in which there are multiple partners.
The Assumptions provide a way to identify significant enabling factors in the Assumptions project’s external context that might influence project success. By making these factors clear and explicit, it will be much easier to monitor whether they are
affecting the project's efforts.
The focus of this handbook is on monitoring and evaluating objectives at the level of development goal, project purpose and outputs. We will also look at monitoring assumptions (which are sometimes referred to as enabling factors).
14 Monitoring and evaluation: tools for biodiversity conservation and development projects
Monitoring the achievement of objectives at the level of activities and inputs will be addressed in a third handbook on Project implementation.
Ss Example:
Assumptions
|. Government is committed to
The unique biodiversity in the urban lowland fragments of the Cape Floristic Region is conserved in a way that benefits people of
biodiversity conservation at high levels and line agencies are required to comply with directives to incorporate biodiversity into planning and to collaborate through the mechanisms established.
: . Government institutions will Development the Cape Flats and is embraced as a valuable ; otal ae ; have the capacity to integrate Goal element of urban life in line with the City . - and implement forward plans of Cape Town’s Integrated Metropolitan ; eneguty ae efficiently. Environmental Policy and Biodiversity a ne . Communities willing to Strategy. Sed engage in opportunities developed in protected areas and sufficient capable civil society organisations are available and willing to undertake C.A.PE. activities. . Funds are raised to support : continued employment of Sustainable conservation management : AS. a ; ; the urban conservation y of sites in the City of Cape Town’s Maen EU Project biodiversity network is achieved through = : : operational costs into the purpose active partnerships between government, nine the private sector and community-based Leeda Reasonably low turnover in organisations. ‘ : community leadership allows capacity to be built. |. Sustainable conservation management is demonstrated at four pilot sites within the City of Cape Town’s biodiversity Outputs conservation network: Edith Stephens
Wetland Park, Harmony Flats Nature Reserve, Macassar Dunes and Wolfgat Nature Reserve.
2. A new cohort of skilled urban conservation managers and champions from surrounding townships is established to conserve the biodiversity of the Cape Flats, and is supported by Cape Flats Nature to implement all other outputs.
Monitoring and evaluation: tools for biodiversity conservation and development projects
. A campaign is conducted to gain broad support for conservation of the four pilot sites and the work of Cape Flats Nature through awareness-raising, promoting use of the sites by the people of the Cape Flats, and securing benefits from biodiversity conservation for the local communities surrounding the pilot sites.
. An advocacy campaign is conducted to secure support for biodiversity conservation on the Cape Flats at all levels of government and lobby, particularly local government leadership, around specific issues as necessary.
. Lessons for sustainable urban conservation management practice from the work of Cape Flats Nature at four pilot sites are captured and shared, and approach of Cape Flats Nature is introduced at two additional sites in the City within the context of a roll- out strategy for the City’s biodiversity network.
16 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Reviewing your objectives
Revisit your plan for your project. Spend 10 minutes reading it through. Consider the following ques- tions:
e Do you have objectives for each of the levels of the project?
The project plan should identify both what you will do (outputs) and what you will accomplish (development goal and project purpose). If you do not have objectives at these different levels, there may be a problem in the design of your change intervention. This could be a risk to your success.
Are you clear on what your Assumptions are?
You do not want a long list of the many factors that might influence your project. But you should be able to identify a small number that you believe could have a significant influence on project success, and that you are not sure will actually occur. These are the Assumptions that you should monitor.
Are your objectives realistic?
You should feel confident about being able to accomplish the objectives with the resources available to you. Taking the external context into account, the objectives should not be too ambitious in terms of: How much you can accomplish (quantity). The standards to which you will be able to do this (quality). The time within which you will achieve these goals (time). The size of the geographic area within which you will work (location).
The various target groups who will benefit from your project (people). Will you be able to measure the objectives?
To collect information on your progress, you will have to be able to measure each of the objectives. While tangible and concrete achievements can often be assessed, it can be more difficult to measure for qualitative components. This issue is discussed in Section 3.
Do you have clear timeframes for the objectives?
You will need realistic deadlines and timeframes for accomplishing the objectives. For those objec- tives that will take some time to be completed, you may have to establish some intermediate tar- gets. These intermediate targets should state what you will have accomplished at particular periods, such as at the end of each six-month period.
Monitoring and evaluation: tools for biodiversity conservation and development projects 17
1.2 WHO WILL USE THE INFORMATION COLLECTED?
Who will use the information collected through your M&E efforts? This is a very important ques-
tion as it will shape your approach to M&E in significant ways. Knowing who your end-users are is important for the following reasons:
e You can clarify the objectives of your M&E work
As you are likely to have different audiences, you may be aware that they often have different needs and interests. You might compile certain information for your own project, but you may have to present the same information in quite different ways to your stakeholder groups. Ask:
« Who wants to know about what we are doing?
# What do they want from us for their interest to be satisfied?
e You can focus your M&E efforts on appropriate information and issues
By being clear about who will be served by your M&E work, you can ensure that you collect appropriate information to meet these needs. This can also save you from wasting time and resources collecting information that is of no interest to any of the end-users.
A further benefit is that you can establish when this information will have to be presented to the end-users. Ask:
# What information do they want?
# When do they want this information?
There are potentially a large number of end-users for information about your project, and each of them may use the information in quite different ways:
* Beneficiaries and/ or the target group may use the information to understand the changes they are experiencing and are part of; to understand the reasons behind decisions that affect the support that is provided to them; and to contribute to their ownership of the project, which will improve the chances of sustaining the effectiveness of the project.
" Project staff can use the results to deepen their understanding of the context of their work; to discover how well they are implementing activities; to find out how the effects of their actions are perceived by others; to understand the reasons for management decision- making; and to learn what problems have to be solved.
« Project and organisational management may use the results to discover reasons for problems; to make decisions for planning and implementing the project; to use the lessons in planning other projects; and for policy-making.
= Donors may use the information to satisfy accountability and reporting requirements; to decide whether or not to continue funding the work; to learn about the effectiveness of different types of projects; and to learn lessons that will influence their support for develop- ment occurring in your sector or country.
There may be other stakeholders that you want to give attention to such as government departments, collegial interest groups and peers, or the general public.
18 Monitoring and evaluation: tools for biodiversity conservation and development projects
TERMINOLOGY/AT/A\GUANCE:
It is common practice to think about accountability only as upwards—to your donors and funders, and to your board of directors. However, accountability can be extended in a number of directions:
» Downward accountability—to those to whom you provide services or who you say you represent.
« Horizontal accountability—to peers and fellow professionals (conservationists)—with a view to meeting certain shared values and standards that are in the interests of the reputation of the sector as a whole.
= Inward accountability—to colleagues, and to the stated purpose and values of the project/ organisation.
To the extent that individuals see themselves as accountable in a variety of ways, this encourages greater self-management and, ironically, allows for more freedom rather than control.
(Cavill & Sohail 2007)
Ss A broad range of information needs...
‘A video will be produced as a marketing tool to promote stewardship throughout the CFR.’ (the beneficiaries)
‘Annual audits are carried out with the farmers who are participating in the programme. There could be more communication with them, on an ongoing basis, about their needs, and perceptions of the benefits of the programme.’ (the target group)
‘Reporting provides information to the funders. Others are kept informed through the database. Everyone feeds information into it and has access to it, so they don’t need reports.’ (project staff and management)
‘Extension staff type the information into the database. The GIS technician checks it and then it is fed through to head office.’ (project staff)
‘The funder’s reporting requirements are programmed into the M&E system. CapeNature’s Scientific Services unit collates the data and produces the required quarterly and annual reports.We also need to provide information required by the host programme (donor and host programme).’
(Conservation Partnerships Programme)
Meeting the needs of all potential end-users may be difficult as it could demand more resources than you can make available. The needs of end-users may also conflict with one another.As you consider the priorities and objectives of your M&E framework, you may have to choose whom to focus on at various points in the project cycle. You might also consider negotiating with users who have some similar interests, to see if you can propose an M&E product that can satisfy their need without meeting all their requirements. Donors and others are willing to be flexible once they understand the cost and burden of reporting, and see that you still wish to be accountable and in a good relationship with them.
The purpose of thinking about the needs of the various end-users is to help you set appropriate objectives for the project monitoring work.
Monitoring and evaluation: tools for biodiversity conservation and development projects 19
Exercise: Clarifying the information needs of project staff
Project staff both generate and use information about the project. It is important to clarify the needs and demands on them, and their expectations in relation to information. Consider their needs in terms of learning, decision-making and accountability.
The following questions may help: What kinds of information do project staff need? Where will they get this information from? How often, and when, is this information required? In what format should the information be made available to project staff?
How will they use the information?
20 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.3 HOW ARE INDICATORS USED AND DEVELOPED?
Indicators tell you what you have to measure to assess whether the project is achieving its objectives. They can range from hard facts and numbers, to changes in people’s behaviour, actions and attitudes.
If you are on a car journey, the road signs allow you to see how close you are to your destination (feedback from the external environment). The odometer informs you how far you have travelled since the start of the journey, while the petrol gauge tells you whether or not you will have sufficient resources to reach your destination (feedback from internal sources). The information can be compared with your plans: ‘After driving for three hours, | expect to have travelled 300 km and to still be 50 km from my destination. | should still have a quarter tank of petrol remaining, enough to reach the destination.’
(Randel 2002)
In a similar way, the M&E process provides both internal and external feedback to project staff.
TERMINOLOGY/AT/A\GUANCE:
According to the WWF (2005), an indicator is ‘a measurable entity related to a specific information need, such as the status of a target, change in a threat, or progress toward an objective. Indicators can be quantitative measures or qualitative observations. Good indicators meet the following criteria:
Measurable: Able to be recorded and analysed in quantitative or in discrete qualitative terms. Precise: Presented or described in such a way that its meaning will be the same to all people.
Consistent: Not changing over time so that the same phenomenon can be measured over time; for example a currency that inflates or deflates in value is not a consistent measure of wealth.
Sensitive: Changing proportionately in response to actual changes in the condition or item being measured.
Quantitative and qualitative indicators
A hot topic within the world of M&E is the difference between quantitative and qualitative objectives and indicators. Particularly in a sector which is steeped in the sciences, the tendency is to place emphasis on what can be counted and quantified. This is made attractive by the many tools that are available for these purposes, for example in sampling and statistical modelling.
An important factor to bear in mind is that quantitative data are based upon qualitative judgments and qualitative data can be described and manipulated numerically (Trochim 2006). Thus, for example, the selection of a particular method for sampling as opposed to another method is a judgment in itself. Similarly, the range of attitudes towards prescribed burning in fynbos areas can be quantified, e.g. the number of landowners in favour and the number of people against. Thus the qualitative and quantitative aspects of M&E are not as polarised as you might have initially thought. There are, however, some helpful distinctions:
Monitoring and evaluation: tools for biodiversity conservation and development projects 21
e Quantitative indicators can be measured through direct observation
Four concise briefing documents about biodiversity conservation on the Cape Flats and its benefits for townships, are developed and distributed to politicians and officials every quarter.
e Qualitative elements may not be easy to observe
You may have to develop specific indicators to assess quality elements.
Did the politicians and officials understand the briefing documents and incorporate them into their work? ( i
e Other elements may be difficult to assess due to technical challenges or the expense of measuring them
You may have to use indirect or proxy (‘in place of’) indicators. Look for indirect measures that will be appropriate in the context of your project.
If there is a clear link between the proxy indicator and the objective you are trying to measure, then you should be able to make informed judgments about the status of the objective.
Is there any reference to biodiversity conservation on the Cape Flats in the development plans on which the targeted officials have worked?
Identifying qualitative indicators
Some tips that may help you identify qualitative indicators:
It is easier to assess behaviour than feelings, as behaviour can be observed.
Ask the question, ‘If the project were a complete failure, how would people know?’ It is sometimes easier to identify indicators for failure. The same indicators, looked at from a positive perspective,
can often be used as indicators of success."VWomen do not participate in public meetings’ might be an indicator of failure. Looked at positively, an indicator of success could be,‘More women partici- pate in public meetings’.
Have a discussion and ask, ‘What do you mean by... (participation, for example)?’ A discussion or brainstorming session can identify how different people understand a concept. By breaking a con- cept into more specific components, it becomes easier to select useful indicators. ‘Increasing the participation of community members in the project’ is a concept that may not be well understood, and for which there may be many different interpretations. By breaking the concept into more spe- cific elements, it can become easier to identify how different project stakeholders understand the concept and its application in the context of the target group and beneficiaries.
(Gosling 1995)
What do the indicators assess?
Each objective is associated with an indicator that helps in assessing different aspects of the project’s progress. While objectives in the project plan describe outputs and outcomes, indicators tell you what progress you are making in achieving your outputs and outcomes.
22
Monitoring and evaluation: tools for biodiversity conservation and development projects
Outcomes
Outcomes indicators assess progress towards the Project Purpose and Development Goal. They are concerned with whether or not the project is making a difference. There are two questions that indicators at an outcomes level are particularly interested in:
« Target group response: Is the target group demonstrating change?
» Are they aware of the project's services and products (its outputs)? » Are they using them in the way intended by the plan?
* Benefits:Are beneficiaries receiving any benefits as a result of the project?
» Is the project making a difference for beneficiaries? » How widely distributed are these benefits?
The project targets conservation managers working on sites in the City of Cape Town’s biodiversity network, by providing them with various forms of support. For its outcome indicators, the project wants to find out:
= Target group response—are the conservation managers who were exposed to the project’s support making any changes in their conservation practice?
* Benefits—are there any benefits for people living on the Cape Flats? Can they be attributed to changes in the practices of conservation managers?
Development goal Indicators Parts of the CFR are conserved in a way that = Increase in priority areas under conservation benefits people of the Cape Flats and is embraced management.
as a valuable element of urban life . nh # : : f f = Increase in baseline jobs associated with con-
servation and nature-based tourism.
Project purpose Indicators Sustainable conservation management of sites is = — Skilled people-centred urban conservation achieved through active partnerships managers are running the sites.
* Local leadership drives community conserva- tion efforts.
(Urban Sustainable Conservation Management Project)
Outputs
The outputs indicators assess whether the planned support to the target group is actually taking place, and how this is being carried out. Its concern with progress is focused on two areas:
= Performance: How are the project outputs being achieved in relation to the plans? Has the project met its targeted goals for:
Monitoring and evaluation: tools for biodiversity conservation and development projects 23
» Quantity—were the specified numbers of services or products delivered?
» Quality—were the outputs delivered to the required standards?
» Timing—were there any delays in providing the outputs?
» Location—were the outputs delivered in the targeted locations?
» People—did the right groups of people receive the outputs?
= Efficiency: Has the project made good use of its human and financial resources?
» Were too many resources used?
» Were too few resources allocated to activities? Did this affect the performance?
The project is interested in output indicators that will help it learn about:
= Performance—what progress has the project made in providing support to the Conservation Managers?
~
= Efficiency—is the project within the budgeted costs for the support it is providing?
Outputs
Sustainable conservation management is demonstrated at four pilot sites
A campaign is conducted to gain broad support for the four pilot sites
(Urban Sustainable Conservation Management Project)
e Enabling indicators
Indicators
Conservation management plans are devel- oped in consultation with the surrounding communities.
Certain activities (alien clearing, flora and bird monitoring, and fire prevention) are carried out in partnership with local communities.
Four community-based organisations with a common values base, are introduced to the project.
The profile of the four pilot sites is raised through regular coverage in mainstream and community media.
A pamphlet and video are distributed for each site.
The enabling indicators assess the current status of the project's Assumptions. These are the ex- ternal factors that are necessary for the success of the project but that are outside the control of the project. In reviewing these indicators, two questions have to be asked:
» Are the external factors that were previously absent, now in place? » Are the important external factors that were present at an earlier stage, still in place?
S
24 Monitoring and evaluation: tools for biodiversity conservation and development projects
The project's enabling indicators help it to be alert to its external environment. Does the external environment support the services that the project provides to the conservation managers? Does the external environment support the conservation managers in carrying out more sustainable conservation management practices?
Assumptions Indicators.
Funds are raised to support continued employment ® Signed contracts and agreements are in place of the urban conservation management team with a number of donors.
There is reasonably low turnover in community = Most of the community leaders in place at the leadership (allowing for capacity to be built) start of the project are involved in the sites
three years down the track.
(Urban Sustainable Conservation Management Project)
How do you develop indicators?
There is no generally agreed method for developing indicators. The previous pages have introduced a number of issues to which you have to pay attention as you develop indicators for your project.
One way of developing indicators involves a three-step process:
|. Make sure the objectives specify what you plan to achieve
Draft objective: Sustainable conservation management.
Specified objective: Sustainable conservation management is demonstrated at four pilot sites within the City of Cape Town’s biodiversity conservation network.
(Urban Sustainable Conservation Management Project)
2. Identify the components of the objective that you want to measure
More than one indicator will sometimes be needed for each objective. You may need indirect or proxy indicators in some cases.
Objective Indicators
Sustainable conservation management is = User-friendly annual conservation manage- demonstrated at four pilot sites within the City of ment plans are developed and implemented at Cape Town’s biodiversity conservation network two sites.
# Infrastructure needs are identified and funded at two pilot sites.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 25
3. Establish how you will collect this evidence
Some of the measures you have identified as your indicators may be very difficult to collect, and you may struggle to gather this information. If this happens, you will not be able to verify (prove) whether the planned changes are taking place, and you should return to the previous step and look for different indicators. If you think you will be able to collect it, then your access to this source of information will become your means to find out what is happening. You can use it to check or verify whether your objectives and intermediate targets have been accomplished.
\))
Objective Indicators Means of (GF Sustainable conservation = User-friendly annual conserva- verification pignazement a demonstrated tion management plans are = Annual conservation man- a four pilot sites oe the developed and implemented at agement plans. City of Cape Town’s biodiversity AVelsies: conservation network = Area manager reports. " Infrastructure needs are identi- fied and funded at two pilot FOR ERS Ts sites.
(Urban Sustainable Conservation Management Project)
In Section 3 we look in more detail at methods of collecting the information.
Some common challenges with developing indicators
e Developing too many indicators
The purpose of indicators is to help you assess the status of your objectives. You need just enough indicators to provide you with this information.
The risk in developing too many indicators is that: * It will be time-consuming to collect all the information you require.
= Some of the information you collect may be irrelevant for the purpose of supporting appro- priate and timely decision-making about the project.
e Impossible measurements
Some indicators may be beyond your capacity to measure. This may happen if: « The objectives are not clear.
" The information will be too difficult or expensive to produce, or is not available to you from other sources.
= The information cannot be collected within the lifetime of the project.An example of this is whether or not beneficiaries continue to benefit from the project's outcomes over an extended period (sustainability).
26 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Measuring the wrong levels
Each indicator should relate to a specific objective. If you try to use the same indicator to measure more than one level of the project, you are unlikely to get information that will help you to manage the project.
Gender-sensitive indicators By now, it is well known that women and men can experience the same situation in different and spe- cific ways. Similarly, your project intervention might target men and women differently.
There are many factors that can be examined to help explain why development impacts differently on women and on men. One factor that is relevant in the context of M&E, is that few projects use gender- sensitive indicators (ones that distinguish between women and men).
The value of developing gender-sensitive indicators can include:
* Including women’s contributions in assessment of the project’s outputs.
whether women are benefiting from the project.
= Assessing whether or not women are carrying an additional burden for household and community work.
If your M&E plan is to include gender-sensitive indicators, it will be important to distinguish between the gender of respondents when gathering information. This is an important step towards ensuring that women are not disadvantaged by development projects.
(CIDA 1997)
r ! ' 1 ! ! ! ' i 1 ! 1 ! 1 1 1 ' ' 1 1 ' 1 ' 1 ® Identifying the adoption of the project results by women as well as men allows you to assess i i : ! f i ! i i ! ! 1 1 1 ! ' t i i i ‘ i I &
The selection of indicators can be a political process. Indicators are subjective, both in terms of how they are selected and how they are interpreted. The use of quantitative, scientific tools does not necessarily make the information more objective and beyond dispute. Laboratory tests to assess water quality may be biased by the localities from which the water was collected, the time of the day or week when it was collected, and by the kind of tests carried out.
It is sometimes necessary to have more than one indicator for each objective. Each indicator
can measure a different aspect of the objective and, together, they can make a more convincing argument about the status of the objective. Indicators are a tool to help you measure changes and to communicate them to people. It would be better to select more appropriate indicators (which stakeholders agree will measure the changes important for managing the project) than to argue whether the indicators are valid ways to assess the objectives.
Monitoring and evaluation: tools for biodiversity conservation and development projects 27
Exercise: Reviewing output indicators
Think of one of the project outputs on which you would like to show progress. Consider the following questions in relation to the indicators for that output:
Do they contain elements of quantity, quality, time, location and people (QQTLP)?
Do they tell you enough about what you have to measure?
Do you have any new information that you can use to update these indicators and will make their formulation clearer?
Do you have a method for collecting the quantitative data?
Do you have a method for collecting the qualitative data?
In the light of these questions, would it be useful to re-formulate these indicators?
28 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.4 WHAT METHODS WILL BE USED TO GATHER INFORMATION?
To have timely, relevant and trustworthy information that you can use to assess whether the de- sired changes are taking place, you will have to use appropriate methods for information-gathering.
If the methods are not appropriate or feasible (including cost, time and expertise), you will not be able to measure the objective. Without any feedback about what is taking place, it will become very difficult to engage in informed decision-making, accountability and learning.
Using legal milestones and technology
‘Our M&E has three aspects to it. Firstly, progress on the number of stewardship sites is monitored by legal tracking, of where contracts are in terms of signing, et cetera. Secondly, the increase in the extent of land covered by stewardship agreements is monitored on a GIS. (These metadata include quite a lot of detail about each site.) Finally, there’s the information that is needed for reporting to C.A.P.E. which is captured on a spreadsheet.’
(Conservation Partnerships Programme)
How can you select methods?
Selecting a suitable method depends on a number of factors. Questions to help you select appropriate methods can include:
e What information is required?
What questions do you want answered? How will the information be used? For problem-solving purposes, understanding why a problem exists can often be more helpful than learning how widespread it is.
e What context and medium of communication will be appropriate?
Some of your information needs can be met by individual informants, while other needs can be satisfied by groups. This individual-group context should inform the methods that will be most appropriate. Additional factors to think about include whether or not the information will be gathered from stakeholders through:
* Written responses (questionnaires).
* Oral means (interviews and surveys).
* Visual means (maps).
* Drama (role plays).
* Story telling.
e What unit of analysis will be used for information-gathering?
The scope and scale of the information-gathering work will also influence the methods you will use. For example, you will need different methods if you are gathering information from individual households in a community compared to gathering information from the community as a whole.
Monitoring and evaluation: tools for biodiversity conservation and development projects
You may also have to use different methods if you are working across a number of geographic areas.
e What resources are available?
While quantitative methods are often more costly in terms of equipment, people and time, qualitative methods require specific skills. Qualitative methods are often more suited to contexts in which time and resources are limited.
Section 3 contains further information on some of the issues and methods involved in gathering data.
Triangulation—using more than one source of information
A challenge that must be faced by all people who collect information, is the question of reliability. How do you know if you can rely on the information that your informants have given you, especially if it sug- gests that you have to make changes to the project plan?
One way to be sure the information is reliable and to reduce bias (whether in the collectors or the informants) is to use more than one method to collect the same information. This approach to cross- checking qualitative information is called Triangulation.
There are different ways of doing it:
# Use a multidisciplinary team to collect the information.
‘The interviews were conducted by a plant ecologist and a social scientist.’
# Use different tools and techniques.
‘In addition to the interviews, a number of photographs were taken of the land that the farmers are utilising, to see how this compared with what they said about the land in the interviews.’
® Collect information about the same issue from different sources.
‘Information gathered during interviews with departmental officials was verified at the group interview with the farmers.’
Triangulation can be especially important when you are assessing the project’s outcomes. It helps you to establish whether the target group is changing as you planned, and whether you have to make changes to your work as a consequence.
(Gosling 1995)
29
30 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.5 WHO WILL PARTICIPATE IN Mae?
Making participation meaningful
‘We have an active multi-stakeholder steering committee. There are 28 members who meet every quarter. It has been really important to make this committee accessible to community members. To make this possible,
we started an Outreach Programme which has three components to it—firstly, we provide opportunities for community representatives to really explore the corridor, to become familiar with the issues. Then we ask them to sit on the committee and to present something from their experience, to bring their experience into the room. Lastly, we set up “knowledge exchanges” where they attend workshops or conferences, and really get to learn from other people. This has made active and meaningful participation possible. If we are not focused enough on transformation issues, for example, Dominee Hans will ask us about this!’
(Biodiversity Corridors Project)
There are many examples of projects in which the M&E work has failed to ring warning bells about looming problems. One reason why this happens is that the M&E plan does not reflect sufficient understanding of the local context. This may become an issue when the plan has been developed and run by outsiders (such as planners or donors) or by specialists (such as researchers and aca- demics).
If the M&E plan is developed and carried out in a way that involves project staff and other local stakeholders, it can bring about a number of benefits.
e Timely and relevant information can be collected from stakeholders
With an improved understanding of the role of M&E in supporting project implementation, stakeholders may be more willing to provide appropriate information.
e The local context can better be taken into account
By paying attention to the social and political context in which the project is being implement- ed, and the value systems in which it is located, information can be more clearly understood in context.
e Any lessons and decisions will be owned and acted on locally
This can have positive consequences for successful project implementation, as staff and other stakeholders understand the reasons behind any decisions to make changes to the project's implementation strategy.
Which stakeholders should participate in monitoring?
Stakeholders will make contributions at different points throughout the project. You do not want to over-burden stakeholders by involving them in gathering and assessing information that has lit- tle direct relevance to them, as it might result in them becoming less willing to contribute as time passes. However, including them in a meaningful way is key to building ownership of the project.
Monitoring and evaluation: tools for biodiversity conservation and development projects 3|
The nature of the relationship between the development practitioner and the stakeholders will determine the quality of exchange—and therefore the quality of information.
You can identify the aspects of the project in which the participation of key stakeholders will be important. You can then assess:
= The relevance for each group of being involved in monitoring; what they can contribute and how it will benefit them.
=» Whether or not they will be end-users of any resulting information; some end-users may want to be involved in gathering the information used to propose changes.
= Whether or not any specialised technical capabilities will be needed. Be clear on the role stake- holders will play in relation to each other in the M&E process so as to maximise participation and not exclude those participants who do not have these technical capabilities.
32 Monitoring and evaluation: tools for biodiversity conservation and development projects
4 Ay Exercise: Mapping the participation process
Think about the different phases in the M&E process. Identify which stakeholders should partici- pate at the different stages of your project:
|. Develop the M&E plan
. Gather the information: » Outcomes. » Outputs.
» Enabling factors.
. Analyse the information
4. Act on the analysis: » Learning. » Decision-making. » Accountability.
(Abbot & Guijt 1998)
Monitoring and evaluation: tools for biodiversity conservation and development projects 33
1.6 WHEN WILL Ma TAKE PLACE?
Although M&E is an ongoing process throughout the project, it is still necessary to schedule spe- cific M&E activities. This can help to ensure that the information will be available when you need it to inform a review of the work, and to support timely learning, decision-making and accountability.
As you think about when you will have to review various aspects of the project, there are a few points to keep in mind:
= If you review progress too frequently, you will end up devoting large amounts of time and other resources to the monitoring process. This can result in the DRIP syndrome (Data-Rich, Infor- mation-Poor).
# If you do not review key indicators often enough, you may miss important information and trends. lf this happens, you may not be able to prepare timely responses to changing conditions.
Many host programmes have a schedule for reporting at regular intervals. If you already have such a system in place, you may be able to adapt it to meet the M&E needs of your project. However, it would probably not be wise to create a new M&E system that you use alongside any existing re- porting system. In addition to the extra resources it will require, it is also likely to create resistance among staff who may feel that they spend more time gathering information and writing reports than implementing the work of the project.
How do you establish the timing?
To establish when you will have to gather information about the various parts of the project, you can explore the following questions:
e Do you need specific information for regular management meetings?
You may have a Project Steering Committee or a similar group that meets to review progress and consider if any changes to the project plan are needed. You should plan to gather the need- ed information far enough in advance of scheduled meetings so that you will have enough time to analyse the information and prepare your recommendations. Care should be taken that it
is not done so far in advance of the meeting that the information is out of date and no longer relevant by the time it is being considered by the decision-makers.
e Do you have to know when a particular activity is completed?
Some of the activities in your project or operational plan may be dependent upon one anoth- er—certain activities cannot take place until earlier ones have been completed.
You can plan to assess if the activities have been finished on schedule, and if the quality of what has been accomplished provides a sufficient foundation for the next step of the work.
34
Monitoring and evaluation: tools for biodiversity conservation and development projects Do you have to reflect on progress at key review dates?
Donors and other stakeholders may require you to submit an annual report, outlining the progress you have made in implementing the project and identifying the outcomes being achieved. This is an important opportunity to reflect on the change that has happened over time, and to gain support from the donor for any changes that might be needed in the plan.As the work is still in progress, you can assess intermediate targets—reviewing whether or not you have made the progress you expected at this point of the project.
When will informants be available?
Reviewing progress on the project outcomes will often require information from representa- tives of the target group and the wider beneficiary population. There are a number of factors that might influence whether or not people will have sufficient time to participate in M&E activities. For example, there are annual financial year-end demands on some staff.
e Are project staff available to carry out M&E work?
The M&E work has to fit into the project schedule along with other activities. You may have to adjust the M&E schedule to accommodate staff as they implement key project activities.
The M&E schedule is often driven by the need for accountability and producing reports. Ideally, it should be driven by the need for reflection and learning, to inform the development approach being implemented by the organisation’s development practitioners or the project team. Once the M&E schedule has been developed, it should be managed like other aspects of the plan by accommodating flexibility. The project will be assessed on whether it achieved its objectives, not on whether it produced all its reports on time.
Use the M&E schedule as a guideline, while placing your focus on the areas where it is needed at the time. It would be better to negotiate changes in report deadlines than to miss key imple- mentation targets.
Monitoring and evaluation: tools for biodiversity conservation and development projects 35
1.7 HOW WILL THE M&£ SYSTEM BE MANAGED?
Managing proactively and creatively
‘I make sure that my projects deliver. This is very important. | am also open and up-front when things don’t go as planned. It is really important to let donors, for example, know when something is not going as it should. But sometimes it happens that you have allocated one month to hire a manager and it ends up taking three months to find the right person. Or the person has to give a longer notice period. But it is always about credibility. About maintaining the credibility of the project.And managing creatively so that you can come as close as possible to delivering on that target. Sometimes this means re-allocating resources—people, time, money.’
(Biodiversity Corridors Project)
The project M&E system that you are developing should be appropriately managed. Along with other components of the project and its management system, the M&E system will need resources and management support to ensure that it is carried out effectively and in a way that contributes to the overall success of the project.
There are a number of problems that can result from a poorly managed M&E system. These prob- lems can harm the project and its reputation among stakeholders. They may include:
" Failing to collect information on time.
= M&E work of poor quality.
= Overspending the budget.
# Limited capabilities.
In addition to the above problems, appropriate management is necessary to ensure that the project
can respond to unexpected developments. The demands of a dynamic context may call for un- scheduled assessments to inform the future direction of the project.
What has to be managed?
As part of the project management system, there are a number of elements that will require atten- tion.
e People = Training and supervision of staff for collection and analysis of information.
= Training of managers to promote the effective use of information for learning and decision- making.
= Supervision to maintain standards of monitoring.
e Systems
= Integration of the information into the management structure. = Meeting data storage, processing and retrieval needs.
= Clear formats and timing of reports.
36
Monitoring and evaluation: tools for biodiversity conservation and development projects
Use of the monitoring information * Learning by staff and partners. = Management responses to unexpected developments.
* Accountability to stakeholders by seeing to it that reports are disseminated timeously.
Time * Sufficient staff time to conduct monitoring activities.
* An early investment of time to build staff capability and stakeholder participation, which is reduced at a later stage.
Financial
= The cost of designing the system.
« The cost of staff time to collect and analyse information. = Resources to print and distribute forms.
* Resources to modify and improve the system.
Monitoring and evaluation: tools for biodiversity conservation and development projects 37
1.8 HOW WILL LEARNING BE BUILT INTO THE Ma PROCESS?
Learning sits at the heart of good M&E practice. It is what transforms the information that has been collected and analysed into new actions and improved practice.
The Action Learning Cycle provides one model for ‘learning while doing’. It focuses on our experi- ences (or the experiences of others), and on reflecting and learning from them.
‘
g R
The Action Learning Cycle
Experience
Planning Reflection
Insight
The elements of the cycle are:
Experience
This comprises the concrete actions and experience of the project. It could be the experience of implementing the work, or it could be information about the context of the project.
= What has happened?
= Who was involved and what did they do?
Reflection
The process of looking back on the experience. This is often done by comparing the experience with what was expected, and looking for similarities and differences.
= ~=Why did it happen? What caused it and what helped it?
= How does it differ from what | expected to happen?
38 Monitoring and evaluation: tools for biodiversity conservation and development projects
Insight and understanding
This occurs as we make sense of what has taken place and become aware of its significance for our plans. This insight can make us wiser!
* What could have been done differently?
# What did | learn? What new insights do | have?
* What new questions do | now have?
Planning new action
! ' U ' ' ‘ i i 1 1 1 ' The insight leads us to consider appropriate responses. By acting on our insights, we can prevent 1 ‘analysis paralysis’ from holding us back. 1
=# What do | want to do?
# What am | going to do differently? t
i
For learning to take place and to result in improvements in the way we do things, we have to go 1 through all four elements of the action learning cycle. Without insights, no new meaning and under- ' : : : 1 standing can result from the reflection, and we may continue to repeat mistakes or be blind to an exter- , nal factor influencing our work.We also have to pay attention to the step of planning, to ensure that we 1 identify concrete ways in which we will do things differently the next time. '
1
'
u
i
i
é
1
a
As soon as the action learning cycle is completed, we commence a new cycle that reviews our latest experiences. Rather than being isolated events, action learning provides us with an ongoing series of linked cycles, or a spiral, in which we are learning and continually improving our approach to our work.
(Taylor et al.1997; Raelin 2000)
The Action Learning Model is one approach to learning. There are many others.
Ss Knowledge exchanges
« | think ‘knowledge exchanges’ are a great way to learn. It gets rid of blinkers. It exposes people to what others are doing, to new things. You can learn from others. The important thing is that you must bring it back and share it. Sometimes you realise that others are not so far ahead of you.This can also be a good learning!
(Biodiversity Corridors Project)
For learning to be possible in a project or organisation, there are certain enabling conditions. One model (Soal, 2007) identifies the following features as being important components of a learning environment:
= Space—creating sufficient dedicated time for reflecting and learning together about your prac- tice.
= Rhythm—regularly coming together to learn so that there is a rhythm in this, something which is compelling and robust.
» Having a champion—someone who has the authority and will to make it happen. This is particu- larly important in the early stages of setting up internal learning.
# Having an approach—a way of learning. This could be through case studies, through presenta- tions, by doing research, having conferences etc.
= Collegiality—a sense that you are trying to build or contribute to a purpose that is bigger than the sum of the individuals who make it up.
Section 2 explores learning further.
Monitoring and evaluation: tools for biodiversity conservation and development projects 39
Exercise: Application of the action learning cycle
Think about the following questions:
" How would you describe the action learning cycle to others? = What are the benefits of this kind of approach to learning?
=# What is its relevance to M&E?
Make notes, and share your thoughts with a colleague.
In Summary,
In Section | we introduced a number of questions to guide you in developing your M&E plan. They were to assist you to:
* Clarify the objectives for your project.
= Establish who the end-users of the M&E work will be.
= Develop indicators to help you assess success.
= Identify methods for information-gathering.
= Identify which stakeholders should participate in project M&E. = Establish the timing of M&E activities.
= Clarify how the M&E system will be managed.
= Recognise the importance of learning in M&E.
The M&E plan guides you in carrying out project M&E activities and processes. It should be flexible, and should be updated as your project plan changes, and as you learn improved ways of working.
The M&E plan can also serve as a tool for communicating with project stakeholders— externally (community representatives, donors and partners) and internally (management and staff). This can inform the stakeholders of the roles that they might be expected to play, and when they can expect to receive progress updates from the project.
REFERENCES AND FURTHER READING
ABBOTT, J. & GUIJT, I. 1998. Changing views on change: participatory approaches to monitoring the environment. SARL Discussion Paper No. 2. International Institute for Environment and Development, London. http://www.iied.org/ pubs/pdfs/6140IIED.pdf (accessed January 2008).
CAVILL, S. & SOHAIL, M. 2007. Increasing strategic accountability: a framework for international NGOs. Development in Practice 17,2.
CIDA 1997. Guide to gender-sensitive indicators. http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Policy/$file/ WID-GUID-E.pdf (accessed January 2008).
GOSLING, L. & EDWARDS, M. 1995. Toolkits: a practical guide to assessment, monitoring, review and evaluation. Development Manual 5. Save the Children, London.
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ.
RANDEL, M. 2002. Planning for monitoring and evaluation. Olive Publications, Durban.
SOAL, S. 2007. Towards ‘better evaluation’-—an account of one internal practice. Keynote address to the Australian Evaluation Society (AES) Conference, September 2007. Melbourne.
TAYLOR, J., MARAIS, D. & KAPLAN, A. 1997. Action learning for development: use your experience to improve your effectiveness. Juta, Cape Town.
WWE (World Wide Fund for Nature) 2005. Basic guidance for Step 2.2: monitoring plan. http://assets.panda.org/ downloads/2_2_monitoring_plan_0|_1|1_05.pdf (accessed January 2008).
40 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 2: LEARNING
In the introduction, we described M&E as a process of ‘learning while doing’. This kind of learning draws directly on our experience. Such an approach to M&E means that change and adaptation are core values and that the insights gained from previous experience are built into future actions. This links closely to the notion of ‘adaptive management’, which is a familiar management tool within the biodiversity conservation sector.
It is this approach to learning that underpins this handbook.
In this section, we focus on the learning that happens in the project team as a way of improving your biodiversity conservation practice and delivering outputs that have greater benefit. This lays the foundation for future steps or phases in the M&E process.
As with other aspects of project M&E, learning can impact on your objectives at two levels:
* At the level of outcomes—the difference the project is making in the lives of the target group and beneficiaries.
* At the level of outputs—the products and services the project is offering to the target group.
Within each of these areas of objectives, there are two questions to consider: = Are we doing the right things? = How do we do things right or better?
By the end of this section, you will have:
* Explored the importance of learning within the context of M&E and some of the ways in which it takes place.
Identified ways of facilitating learning within the project team.
* Amore in-depth understanding of the case study method and how it could be applied in your con- text.
a 1 f 1 1 1 1 1 1 t i 1 1 # Ideas for building a learning environment. 1 |
Monitoring and evaluation: tools for biodiversity conservation and development projects 4\
2.1 HOW DOES LEARNING HAPPEN?
A useful learning concept is the notion of single- and double-loop learning. It draws our attention to the importance of un-learning and re-learning, and some of the barriers we might experience to new ways of doing things. These barriers are often experienced as challenges to our existing ‘men- tal models’.
Challenging our mental models
What happens when we come from different places
‘One of the challenges for us was around entrenching an external project (which had been operating for a while and had its own way of doing things) into an existing organisation (which also had its own culture and way of doing things).We thought the business units were slow in understanding and incorporating the project’s ideas and approach into their operations. They probably felt the same way about us! An “us and them” mindset seemed to get in the way.
(Conservation Partnerships Programme)
This example illustrates the power of our mental models at work. Mental models, or paradigms, are the ways in which we understand how the world works. Our repeated experience of seeing certain things happen again and again, makes us form a mental model that tells us that such things will always happen in this way.
The project staff had a mental model for how the project should be implemented. The staff at the busi- ness units had a mental model for how they approached their work.At some sites, even though the business unit staff were aware of the achievements that the project was showing elsewhere, they were unable to overcome their ‘paradigm blockage’ because they were attached to their standard way of thinking and could not integrate the new ideas into their work.
When we are analysing the M&E information, we are at risk of experiencing paradigm blockage.As we look for the causes for the changes we are being made aware of, we may find ourselves limited by the boundaries of our mental models.We may not even know we are being limited because it is very dif- ficult to be aware of what forms our mental models.
We may be able to challenge our mental models if we consider how we look at the information that lies before us.
e Single-loop learning In this kind of learning, a response to an observation or event is based on a person's (or organisation’s) existing set of values, beliefs, norms. If the activities we embark on do not produce the results we expect, we change the activities in the hope that it will bring about the desired results. This is an ‘error correction’ approach to learning.
e Double-loop learning
In double-loop learning, our responses emerge from a reflection on dominant or stated values, beliefs and norms. The information that things have changed provides a challenge to the thinking that lies behind our current activities.As the activities did not bring about the expected results, we question our assumptions and the mental models that led us to these activities in the first place. We
re-think the rules, so to speak.
Fr 4 i 1 i I 1 1 1 i 1 1 1 I 1 1 1 i 1 i 1 i I I 4 i 1 1 Q i i : i 1 4 j fl i 1 I 1 1 1 1 i 1 i f 1 ' 1 1 1 I i i ‘ 1 I I A aaaes : : : 1 The value of double-loop learning is that it helps us to step back from our habitual reactions to
\ have a critical look at our mental models. This may lead us to question the underlying assumptions that inform the project’s plans.
i
42 Monitoring and evaluation: tools for biodiversity conservation and development projects
~~" sss se se es se eet erhlUCcrrhlUlCUcrTClUC rClClCUcrClCcrlClClCcrrlClCchrlhlClCcrlClCchOrlClCchrrlClCcrrlhlClCcrOrlhlClCcrrlhlClCchrrlhlCchrrlhlCcrOlhlCUcrhrlUcrrlClCcrlClCcrlClCcrlhlCchl rT OC rOllCUcrOlClCcrlClCcrlClCcrrlClCcrrlClUCrrlClCUcrhrlUcrlUcrlUcrlU | Two other concepts that are similar to single- and double-loop learning are patterning and puzzling:
e Patterning
In this process of learning, we develop meaning by linking our current experience with similar ones we have had in the past—this is learning by generalisation.
e Puzzling
Puzzling happens when we are confronted with an experience that we have not had before or that contradicts previous experiences—this is learning by exception, contradiction or surprise.
(Raelin 2000; Smillie & Hailey 2001; Uphoff & Combs 2001; Rogers & Williams 2006)
There is a place for these different kinds of learning, including single-loop learning and patterning which, for example, enable us to become adept in a particular skill. It is when we are stuck that
we may have to turn to other ways of learning. While double-loop learning is not necessarily more time-consuming, it can be much more challenging than single-loop learning. It asks you to look criti- cally at your basic beliefs and assumptions about the project’s goals and context as well as at your current way of working. ‘Puzzling’ is similarly challenging.
Helping people to learn, with these concepts in mind, requires skilled facilitation. Managers must strive to find a balance between support and challenge in helping people to let go of old ways of thinking and doing, and taking on new ones.
In your role as a project manager, you may find it useful to bear this in mind when working with people in learning situations.
Monitoring and evaluation: tools for biodiversity conservation and development projects 43
2.2 LEARNING IN THE PROJECT TEAM
Encouraging learning in the project team is a key issue. By helping project members to develop new insights and understanding, you will be able to contribute to improved work practices and hence to improved outputs that you can offer to the target group.
Learning can come from many different directions. Our day-to-day experience in the field is an important source of learning. So is the wider context within which we implement this work.
Case studies are particularly valuable for looking at the experiences and practices of staff ona daily basis. This is the tool on which we focus here. Some of the other ways in which the team can learn are described briefly. They are: after action reviews, peer assists, debriefing, team meetings and workshops, and benchmarking.
Exercise: Trying a new tool
. Can you think of a recent question or problem within your work environment with which you have been grappling that would benefit from being presented as a case study?
. Spend 20 minutes writing it up, taking into account the following questions: What happened? Who was there? What did you do? What were you thinking and feeling at the time? What were the results of what you did?
What new questions and thoughts do you have?
. Identify three or four colleagues who have a similar commitment to learning as you do, and tell them what you know about case studies (see below).
. Ask them if they would be willing to try out this new tool with you. . Agree different roles for the first session.
. Decide on a time and place.
. Debrief at the end of the first session, and test if there is any interest in trying it again.
Using case studies to learn about your conservation practice
e What are case studies?
Case studies are particularly useful for gathering information on your current work practices,
for describing what you are doing in the field and for learning about what works and does not work. Using a case study is an opportunity to present a persistent question or problem and to explore and solve this problem with others.
44
Monitoring and evaluation: tools for biodiversity conservation and development projects
Case studies, described in this way, contribute to the monitoring of performance-related as- pects of a project’s outputs. They help to answer the questions “What progress is being made in providing goods and services to the target group?’ and ‘What are the standards to which they are being delivered?’ Case studies can also reveal any delays there might be in delivering these services. They can deal with any aspect of quantity, quality, timing, location and people (see Sec- tion |.3).
There are four components to a case study
* Getting into the experience—reminding yourself of a particular situation, what happened, what you did, thought or felt, what the results were and what new thoughts and questions arise.
* Describing this experience by writing it up as a case study. « Sharing the case study with colleagues by presenting it. * Having a discussion in which colleagues listen, ask questions and provide input, generating
new perspectives and angles on the issue in question.
The skill of listening
Central to all of these components, is the skill of listening. It requires that, as a colleague, you are able to hear what the presenter is saying without judging it. This kind of listening is demand- ing—it is listening with the purpose of understanding the whole story and what the presenter is telling you. There are three levels at which listening has to happen in order to hear the whole person:
= At the level of ideas, thoughts and facts. = At the level of feelings, reactions and emotions.
# At the level of will and intention.
Most of us are used to giving advice, to being an expert, to having the answers. When we work with our colleagues in a listening mode, it is less about us and more about them.
In running a case study session, there are three key roles:
Chair and time keeper. Purpose is to:
# Decide how long you have for each case study (including presentation and discussion) and to keep to the time allocation.
= Ensure that the presenter of the case study is not interrupted.
= Encourage participation and give space for discussion, being careful that one person does not dominate.
* Try and follow the threads of the discussion.
= Summarise the discussion and a possible way forward.
Case study presenter. Purpose is to:
= Read the case study and the questions, slowly and clearly.
= Bring a level of distance to the discussion and to be open to the outcomes of the discussion. In doing so, he/she does not have to respond to every point and to defend his/her choices. He or she should allow the discussion to flow.
* Really listen to what his/her colleagues are saying.
Monitoring and evaluation: tools for biodiversity conservation and development projects 45
Note-taker and reminder of why we have case studies. Purpose is to:
= Remind everyone at the outset to listen at the different levels, and to work at really hearing and understanding the presenter’s story.
= Remind people not to judge language or style.
* Take notes of the discussion.
These notes are important in generating information for more conventional forms of learning and sharing, such as report writing to funders, for annual reports et cetera.
The benefits of using case studies
As a result of this process, you will have generated valuable lessons on what is and is not work- ing for both the individual practitioner and the project as a whole. This can lead to important improvements in the delivery of goods and services to the target group.
After Action Reviews
This is a way for a team to reflect on a piece of the work that has recently been completed, and to draw out lessons. It can be used by the whole team as well as by subteams and individuals. (This is elaborated in the box below.)
Peer Assists
A team starting a new project or preparing for an event in which they have limited experience can call on people with experience for help. They can be invited to join the team in a meeting or a workshop in which the guests tell their stories and share their insights and hypotheses. The team’s aim is to learn from their peers about appropriate ways to approach the tasks lying ahead.
Debriefing
This is a structured way for an individual or a team to reflect on their experience and to share their insights with the rest of the team. It is best to do this at the end of an event or a phase of work, as it allows the whole process to be reviewed. Debriefings can easily be recorded or written up, allowing them to be shared with a wider audience at a later stage.
Team meetings and workshops
Any time that the team gather for a meeting or a workshop, there is an opportunity for learn- ing. Part of the meeting can be used for reflecting on past events, making meaning of these experiences, and drawing out insights and hypotheses for future action.
Benchmarking
This is a technique for learning from the experience and ‘good practices’ of other organisations. It can be especially appropriate when you are moving into a new area of work or are trying to improve the standard of your work. By looking at how other organisations do things, you can identify the ‘benchmark’ or standard for the way you will do things.
46 Monitoring and evaluation: tools for biodiversity conservation and development projects
After Action Reviews
The After Action Review (AAR) is a tool for team-based learning.A team can use this tool on a regular basis to develop insights into their experience and to make meaning of the information they are gathering through their monitoring. The benefit of the AAR is that it does not require a large investment of time or other resources.
The process for an AAR involves reflecting on a meeting, workshop or any project activity that has just finished. Members of a team ask themselves:
=" What was supposed to happen?
* What actually happened?
=" Why was there a difference?
# What will we do differently next time?
The AAR allows the team to engage in ongoing reflection and continuous improvement without waiting
for formal monitoring to take place. Lessons can be identified and acted upon by the team, or passed to others in the project or organisation where appropriate.
The AAR is an abbreviated form of the project monitoring process. It poses questions similar to the ones asked during project monitoring.
By asking ‘What can we do differently next time?’ you can identify situations in which the lessons you have just learnt can be relevant.
e Incremental improvements
If you are running a series of related activities, you can improve the way you approach later activities by including any lessons from the earlier events.An example would be if you were running a series of workshops for a group of people.An AAR will help you to identify ways to make each workshop better than the previous one.
e Other related opportunities
The AAR lessons can be applied to situations that are not the same as the one you have just finished, but for which the lessons are relevant. Lessons from one workshop can help you to improve an unrelated workshop you are running for a different audience, or it may help you to identify ways to publicise your work better.
(Collison & Parcell 2004)
G When learning becomes a habit
After each of our events or activities, such as Water Week or Dunes’ Day, we evaluate what worked or did not work. We write this up and when we meet again, to plan another activity, we bring these notes and we use them to plan better for the next event.We go back to these notes when we do our future planning. We are now in the habit of doing these reviews cnd they really work.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 47
improving development and conservation practice
As mentioned in the introduction, one of the key M&E questions is, ‘How do we do things better?’ This goes to the heart of our development and conservation practice. One way of improving our practice is to identify the level at which we are currently performing (the ‘condition’), and then seek ways to improve this by identifying the ‘pathway’ that is appropriate for that condition.
Pathways to doing things better
1. You do not know if performance is good or bad.
2. Performance is poor but you do not know how to improve it.
3. Performance is poor but you lack the capacity to improve it (resources or authority).
4. You know performance is poor but there is a lack of incentives for improving it.
5. You are doing well but you do not know how to maintain or replicate it.
(Rogers 2007)
Seek feedback or information immediately and more regularly.
Seek expert knowledge or advice about better ways to do things (knowledge building).
Seek general support through advocacy.
Use carrots, sermons or visioning (use incentives).
Learn from your success and how it can be replicated, particularly through the process of documentation.
To integrate learning into your work, the process of making changes requires the support of the various stakeholders. This can help to increase the likelihood that the target group and donors, for example, will provide their backing for the new measures, and that any necessary resources will be
made available.
It is helpful to keep in mind that the lessons you are formulating represent insights and hypotheses. They are your interpretation of what has taken place, of how you have made sense of your experi- ence. It is therefore useful to approach any planned changes as experiments. By treating them in this way, you can pay close attention to what happens. If necessary, you can revise your hypotheses over time, adding new information to confirm them or change them as needed. This will deepen
your adaptive management approach to M&E.
48
Monitoring and evaluation: tools for biodiversity conservation and development projects
introducing changes in your practice
The process of introducing change may occur in a variety of ways (Fowler 2000):
Continuous improvement and tinkering
The project makes minor changes as a response to learning from its experience. It can become a key aspect of the project team’s culture over time.A risk you should be alert to, is that tinker- ing with small changes may blind you to the need to question some of the central assumptions
held by the project (see the earlier subsection: Challenging our mental models).
When the project first started providing support to the conservators, they met with them often and were very active in the work of the conservators.
This met with some resistance from the conservators who felt that the project was intervening too much.We cut back on the frequency of the meetings.
(Urban Sustainable Conservation Management Project)
Reflective evolution
Changes can arise as a consequence of reflection becoming central to the project team’s way of working. The stimulus for change is the reflection, not a crisis or the periodic monitoring reviews. Reflection encourages a more formal recognition of what staff may already be feeling intuitively. lt promotes proactive changes and renewal of the project strategy.
Over time, as the conservators became confident in giving feedback and the project’s management were able to listen to it, the meetings with the conservators became more focused and purposeful. The project trained the conservators in the practice of developing and presenting case studies. The conservators have found this type of meeting to be very helpful. It provides them with support in the field and it creates a safe place for them to share and discuss their challenges and problems.
(Urban Sustainable Conservation Management Project)
Process reform
The project may make changes to the way it goes about its work, without changing the role it
is seeking to play. Changes to the process may often occur in response to major changes in the external environment. The project might stop providing certain services if it duplicates what another organisation is doing. The project can then use its staff and resources to focus on other, still neglected services.
While the project still aims to contribute to people-centred conservation management, local government has now become better at supporting the conservationists that it employs. The project has stopped facilitating and running the meetings with the conservationists. Instead, these are driven by the City.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 49
There are further possible degrees of change, such as role changes, restructuring and transformation. These changes are more likely to involve far-reaching shifts in approach, structure and staffing, and may not be relevant within the context of projects. These options, however, might be relevant for organisations that manage projects as part of their work.
50
Monitoring and evaluation: tools for biodiversity conservation and development projects
2.3 BUILDING A LEARNING ENVIRONMENT
The context in which learning happens is integral to its effectiveness. In addition to the factors that were identified in Section |, there are a number of other considerations in building a learning envi- ronment. Roger Harrison (cited in Thaw 2002) identifies six factors that encourage learning.
There is constant attention to the learning process
The project or organisation values and encourages learning. People within the project team see it as valuable.
There is an interest in finding longer-term solutions
Learning is not only aimed at solving immediate problems but attempts are made to understand the bigger picture, as well as to distinguish symptoms from deeper causes.
Learning is self-initiated and self-directed
Staff are encouraged to take responsibility for their own learning and do not rely on others to drive and identify learning opportunities.
People are able to think systemically
Because the situations in which people are working are complex, they must be able to focus on the complexity of relationships, and on the whole, rather than on the parts.
Questions, questions and more questions
Learning works with questions, and questioning certain assumptions, values and practices of the project or organisation. It does not ignore certain patterns—nothing is ‘un-discussable’ and there are few ‘holy cows’!
There is a system in place to support the learning
This system enables the project or organisation to take in complex new information and modify itself. It also has policies and processes in place to support individual learning efforts.
There are two common barriers to learning that project managers should keep in mind:
The presence of fear, anxiety and other strong negative emotions in the project team or organi- sation.
The bias-for-action, which is a characteristic of most leaders and managers, and is prevalent in the culture of the development sector.
Monitoring and evaluation: tools for biodiversity conservation and development projects 51
Exercise: Learning in your project
ca
Taking into account the factors identified above:
= Where do you see learning happening in the project? What and who are making it possible for learning to happen?
Where is learning not happening? What and who are preventing learning from happening?
Why do you think this is so?
What do you think you can do to support and encourage learning within the project? How could you influence it?
Do you have to take any steps to deal with negative emotions in the project (such as fear and anxi- ety)? What steps could you take?
Do you have to take any steps to deal with a ‘bias-for-action’ within the project? What steps could you take?
In| Summary,
In Section 2 we:
= Explored some of the concepts underpinning learning. " Focused on learning within the project team by: » Identifying the different sources of learning.
» Discussing some of the methods and tools that encourage learning, in particular active reflection on own and others’ experiences.
» Identifying ways in which to introduce changes in professional practice. " Looked at the use of case studies as a tool for capturing learning.
" Identified components of a learning environment.
REFERENCES AND FURTHER READING
COLLISON, C. & PARCELL, G. 2004. Learning to fly: practical lessons from one of the world’s leading knowledge companies. Capstone, Oxford.
FOWLER, A. 2000. The virtuous spiral: a guide to sustainability for NGOs in international development. Earthscan, London. HARRISON, R. 1995. The collected papers of Roger Harrison. McGraw-Hill, New York. RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ.
ROGERS, P. 2007. Building evaluation capacity and capability in organisations. Open day held at CDRA Offices, Cape Town, 23 November 2007.
ROGERS, PJ. & WILLIAMS, B. 2006. Evaluation for practice improvement and organisational learning. In |. Shaw, J. Greene & M. Mark (eds), Handbook of evaluation. Sage Publications, London.
SMILLIE, |. & HAILEY, J. 2001. Managing for change: leadership, strategy and management in Asian NGOs. Earthscan, London. THAW, D. 1996-2002. Ideas for a Change Series. Olive Publications, Durban.
UPHOFF, N. & COMBS, J. 2001. Some things can’t be true but are: rice, rickets and what else? http://www.eldis.org/ fulltext/TRUE-RRR.pdf (accessed January 2008).
52 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 3: GATHERING THE INFORMATION
Information is the fuel that drives the project M&E system. It makes it possible to know what has happened and what challenges may be looming on the horizon.A well-designed M&E system will enable you to collect appropriate information on the work and.on the project’s context in a way that will feed into timely and relevant decision-making, reporting and learning.
There are three broad considerations in the information-gathering process:
* What is the nature of the information needed for M&E purposes?
«= What are appropriate methods for gathering the information?
= How can you ensure that the information is accessible and useful for evaluation purposes?
A useful starting place is to refer back to the project plan. These plans provide a description of what should be monitored (i.e. the indicators) and from where that information should be sourced
(i.e. the means of verification). They also remind you of what you said you were going to achieve. The frequency of measurement may also be stipulated in the plan.
A further consideration is deciding who will be involved in providing and collecting information. This has to be co-ordinated and managed.
By the end of this section, you will have:
« A better understanding of the kind of information needed to measure objectives at the level of outcomes and outputs.
= Compared different methods for collecting data, and started to select methods appropriate to your context.
" Been provided with some ideas on ways to manage collective information so that it is useful and accessible.
Often when we are monitoring, we are counting the ‘blomme in the veld’. This is about numbers, about what is quantifiable. We look at the number of hectares under conservation, how genetically different the populations are. This provides us with rich detail. It helps us to be succinct in our communication.And while this is necessary work, it is not sufficient. Sometimes we have to look at the relationships, to see what is going on there, why things are not happening. This is much harder to measure.We use case studies to bring these issues to the fore.
Mostly the information is in our heads or in files. We do not have to go and gather extra information. But the process of extracting it can be laborious. It takes time.And making phone calls.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 53
3.1 UNDERSTANDING YOUR INFORMATION NEEDS
Your search for information will be driven by the indicators you have developed (indicators are dis- cussed in greater depth in Section |) in your project plan. They will direct you to the information you will need to collect in order to assess what progress has been made for each of the objectives you are monitoring.
e Information relating to outcomes
Information for monitoring outcomes tells you whether the project is having any positive influ- ence on the target group and beneficiaries. It is information that is used for comparative pur- poses. It usually compares the state or status of people or things over time and, sometimes, also between different groups or study subjects at the same time.
Development goal Indicators « Parts of the CFR are conserved in a way = Increase in priority areas under conservation that benefits people of the Cape Flats and is management.
embraced as a valuable element of urban life g het 25 : : Increase in baseline jobs associated with con-
servation and nature-based tourism.
(Urban Sustainable Conservation Management Project)
As indicators are typically used to measure the extent of changes that have taken place, it is often nec- essary to know what situation existed at the start of the project. This starting point is called the base- line and provides a point of reference against which to measure the changes. Baseline data can be col- lected by quantitative or qualitative methods, depending on the kind of information you want to gather.
One problem that many projects face is that they do not have a baseline reference point, and so they are unable to assess the extent of any changes that are taking place. Baseline information should ideally be collected during the project planning process, and should form part of the rationale for establish- ing the project. It is possible, however, to overcome the absence of such information by carrying out a baseline study as one of the early activities of the project.
It can be complex to gather baseline information, and it can be a real challenge to know how typical the measurements of the usual state of affairs are. In some projects, it may be more useful not to waste resources on a baseline study. Instead, changes can be identified by comparing the first assessment with later assessments.
(Feuerstein | 986)
e Information relating to outputs
Outputs are the results of activities undertaken during the course of the project. Output infor- mation generally relates to the delivery of certain goods, services and products, and provides answers to the questions, ‘Is the project making progress towards its objectives?’, "What is the progress?’, and ‘How efficiently is the project using its resources?’ The information tells you not only whether progress is being made but also about the quality and scale of the achievement.
54 Monitoring and evaluation: tools for biodiversity conservation and development projects
I outputs Indicators A campaign is conducted to gain broad support " Four community-based organisations, with a for the four pilot sites common values base, are introduced to the project.
= The profile of the four pilot sites is raised through monthly coverage in mainstream and community media.
= A pamphlet and video are distributed for each site within a year.
(Urban Sustainable Conservation Management Project)
As the outputs are dependent upon the internal activities of the project staff, much of the monitoring information will be available through internal project documentation and other sec- ondary material sources.An ongoing challenge for all projects is to verify that what is described in the documents is what has actually happened. For example, staff may be tempted to inflate the number of people who attended community meetings in an effort to please management and donors by showing that there is strong community support for the project.
e Information relating to assumptions
In gathering information about the assumptions, you are looking for the presence or absence of certain conditions. These conditions are outside the control of the project, but are critical to its success. Information about the assumptions acts as an ‘early warning system’, drawing attention to potential risks to the success of the project.Assumptions are frequently overlooked in an
M&E plan. Fs C Assumptions Indicators Funds are raised to support continued = Signed contracts and agreements with a employment of the urban conservation number of donors are in place.
management team.
There is reasonably low turnover in community = Most of the community leaders in place at leadership (allowing for capacity to be built). the start of the project, are involved in the sites three years down the track.
(Urban Sustainable Conservation Management Project)
As discussed in Section |, the selection of an appropriate method also depends on whether the information that you need is qualitative or quantitative.
Monitoring and evaluation: tools for biodiversity conservation and development projects 55
e Qualitative information
This is information that tells you about human or social behaviour and perceptions. Qualitative research generally provides answers to ‘How?’, ‘What?’ or ‘Why?’ questions and is recorded in narrative format, for example as stories or case studies.
e Quantitative information
Quantitative information generally provides answers to questions of ‘How many?’, ‘When?’ and ‘Where?’. This is the type of data scientists are most familiar with. lt may be presented in the form of statistics or graphs. Often it is percentages or other relational measurements, as op- posed to numbers, that hold significance as indicators.
‘In this project, we have to maintain a certain number of kilometres of road.’
This is quantitative information that can be collected by measuring the length of the road on which work has been done.
‘What we have also realised is that the roads have to be maintained to a certain standard. One expert came and told us that when the rains come, these roads will be washed away.’
There are certain technical standards for road maintenance that can be used to establish the qualitative indicators for this output. They tell us the ‘how’ aspect of road maintenance.
(Project Developers’ Forum, |8—19 March 2008)
The type of M&E that we are discussing in this handbook, what we referred to in the introduction as ‘effectiveness measurement, is likely to require a combination of both qualitative and quantitative information.
You should also find out if you will have to collect the information yourself (primary information) or if the information is already available in a format that you can use (secondary information).
e Primary information
This is information that you will be the first to collect. Selecting appropriate methods, you will gather this new information directly from project participants/stakeholders, and from the target group and project beneficiaries. You may even gather information from the general population in which the target group is located, especially for information to help you assess outcomes indicators.
e Secondary information
This is information that already exists, collected by other people or other organisations. Unless it is material existing within your own organisation, it is likely that you will have access to this information only in its completed form—as reports and articles, rather than as survey respons- es and interview notes.
56 Monitoring and evaluation: tools for biodiversity conservation and development projects
Generally, gathering information from secondary material is likely to be less costly than undertak- ing primary research. In some cases, secondary information will be sufficient to meet your needs.
In other cases, there may not be any secondary material available, or it may not be sufficient or suitable for your information needs. You will have to select an appropriate method to conduct the research that will provide you with the required primary information.
In considering the options, remember that you are looking for indicators, and indicators are just that—they provide an indication of the state of things. They are not intended to be exact measure- ments. They provide an indication of a shift or change. It may be impossible, for example, to do a head count of every bus load of visitors, but a record of the number of buses could be maintained and an estimate of the number of visitors extrapolated from that.As a guiding principle, some information is always better than none, no matter how unscientific it may seem, as long as it is consistent.
Monitoring and evaluation: tools for biodiversity conservation and development projects 57
3.2 SELECTING AND USING APPROPRIATE METHODS
Depending on the nature of the indicator, a wide variety of methods are available for information gathering purposes. For example, interviews are useful for gathering information on people’s per- ceptions and views on a particular issue, whereas a camera may help to capture hard evidence on the physical condition of something. The challenge is to identify relevant and appropriate methods that will help you get the information you need to analyse progress.Appropriate methods can be selected with the use of relevant criteria.
Each discipline and development sector has specific methods that can be used to help gather relevant information for technical monitoring purposes. For example, agriculturalists have ways to assess soil quality, health practitioners have ways to test for the presence of health problems, and conservation planners have ways of recording changes in the extent of a particular type of vegeta- tion over time.
In this section, we aim to expand your repertoire of monitoring techniques by introducing tech- niques that are useful for gathering information from and about people and the social contexts that impact on their behaviour.
Criteria for selecting appropriate methods e Timelines
Will you be able to gather the information within the necessary timeframe? If the method takes too long to prepare, you will be unable to meet your reporting deadlines. If the method is very time-consuming to carry out, you may not have enough staff to continue with project activities as well as conduct the monitoring.
e Relevance
Will the method provide you with the information you are looking for? If the information does not directly answer your information questions, you will have to reach your final answer through guesswork! As this is not a strong basis for making important decisions about the fu- ture direction of the project, it may be necessary to search for a more relevant method.
e Cost-effectiveness
Will you be using an appropriate level of resources to gather this information? Your monitoring budget should be distributed appropriately across the range of monitoring activities. If a meth- od is too time-consuming, or requires a large investment in training or specialised equipment, it may not be as cost-effective as an alternative method to gather the same information.
There may be additional criteria that you can identify to help you select methods to use in your project monitoring process.
It is important to keep in mind that project monitoring is done to meet the practical informa- tion needs of the project. Practical, easy-to-use and reliable methods that will deliver the infor- mation are preferable to complex ones designed to investigate research questions that deliver little practical benefit to the project.
58
Most useful for information
Monitoring and evaluation: tools for biodiversity conservation and development projects
Techniques for gathering information
Other factors to consider when selecting
the most appropriate technique
Secondary material and documents
Individual interviews
Group interviews
Key informant interviews
Storytelling and focus groups
Critical event analysis
Participant observation
Questionnaires and surveys
Forms
Diaries and logbooks
Photographs and videos
Calendars
Daily and weekly routines
Mapping
Ranking
Rich source of information that is readily accessible.
Allows confidentiality. Time consuming but can be a rich source of qualitative information.
Capture information from up to 10 people in a short time. May not reveal sensitive information but may be more detailed.
Useful for gathering objective views and ‘second-hand’ perspectives.
Useful for gathering information on a specific topic—generates dialogue and reflection. Rich in anecdotes that carry outcomes indicator information.
Useful information for diagnostic studies where a problem is being investigated or for reconstructing the way it was solved.
Time consuming but useful for gathering information about the way people behave when going about their normal tasks without project staff present.
Useful for collecting quantitative information from a large number of people.
Useful for collecting information from a large number of people at regular intervals, and that can be used for comparative purposes.
Useful for collecting quantitative and qualitative information that can be used for comparative purposes.
Useful for recording changes over time for comparative purposes.
Useful for identifying cyclical patterns or links between behaviour and the seasons.
Useful for periodic, rather than regular, collection of information using forms or interviews.
Useful for collecting spatial information on the indicators being monitored.
Useful for collecting information on priorities or preferences where there are options involved.
Monitoring and evaluation: tools for biodiversity conservation and development projects 59
Exercise: Methods for collecting information
. Think of a question, relating to your project’s target group, that could be discussed in an interview situation. Read the descriptions of the various types of interviews in ‘A toolbox of techniques for gathering information’, including Critical Event Analysis.
. Consider the pros and cons of each interview method, relative to the question that you chose. Use the summary table of ‘Techniques for gathering information’ to help you decide which would be the most appropriate.
A toolbox of techniques for gathering information
e Secondary material and documents
Much of the information you seek may already exist in documents and this can save you from dupli- cating work. For example:
= Project records—forms, logbooks, budgets and financial reports.
* Supervision checklists and reports.
=" Correspondence.
=" Media—articles published in newspapers and videos of television reports.
= Statistics, documents and reports published by other organisations, such as government depart- ments and other NGOs, for their own use.
Secondary material can help you to see patterns and trends that you can use to inform the ques- tions you might use in interviews and surveys. They can also help you to select other appropriate methods to investigate particular issues in more depth.
e Individual interviews
The confidential nature of individual interviews often allows interviewees to discuss issues more openly and they may reveal more than they would in a group setting.
The interview is best conducted in a semistructured way, with open-ended questions being used to collect information, experiences, descriptions, opinions and perspectives. You can ask the interview- ees to clarify their answers and to expand on interesting views. By using similar questions with a range of interviewees, you can bring out a range of experiences and views about similar events.
e Group interviews
Group interviews allow you to gather information about the experience and perspectives of a larger group of people than is possible with individual interviews. Such interviews are often ho- mogenous—they bring together people who have had similar experiences, or are part of the same group. However, groups should not be too large, perhaps no more than 10 people. If necessary, subgroups can be formed.
Group interviews are less likely to reveal sensitive information than individual interviews. However, interviewees may provide more detailed information as they interact with one another and check with one another on facts. Group interviews may need careful preparation on the part of the interviewer so as to ensure that all the members have a chance to express themselves during the interview. If the interview is dominated by a small number of members, it may be possible to hold informal conversations with quieter members at the end of the interview.
Monitoring and evaluation: tools for biodiversity conservation and development projects
e Key informant interviews
You can interview people who are specialists on relevant topics for their views on the way the project is influencing its target group. Such informants can be more objective as they are not personally involved in implementing the project or benefiting from it. Key informant interviews are particularly appropriate if they can provide information and perspectives not available to the project staff.
They can be asked to provide information on the experiences and perspectives of others, such as the target groups, assuming that their contact with the target groups has allowed them to witness what they are being asked to describe. It will be important to verify these ‘second-hand’ perspectives with information received ‘first-hand’ from project beneficiaries and target groups. Open-ended questions can allow them to describe things that may not have occurred to the interviewee to ask.
Storytelling and focus groups
These groups are convened for a detailed discussion on specific topics.A variation on group interviews, they bring together people who have had particular experiences, or who have an interest in the topic. The interviewer (who serves more as a facilitator) asks few questions, and encourages participants to provide longer, more story-like, responses.As participants respond to one another, a dialogue can develop. The facilitator can guide the discussion from the sidelines. By asking reflective questions, the facilitator can help the group to explore patterns and themes.
Critical event analysis
This method is a way of focusing interviews with individuals or groups on particular events or incidents. The purpose of doing so is to allow interviewees to reflect on as many of the ele- ments and processes that they can describe, so as to create a richly textured understanding of the event.
Participant observation
This method allows direct observation of events, processes, relationships and behaviour by the researcher. It can be a useful way to verify whether the information provided through interviews and surveys can be trusted or not. It can also be used to collect information to help design interview and survey questions.
Observation is best carried out by someone unknown to the observed group. If a fieldworker familiar to the group was to do this, his or her existing perspectives and pattern of interactions with the group may prevent him or her from observing some significant things. The best insights can often be obtained by a researcher who joins in tasks with the group, becoming a ‘participant observer’.
Questionnaires and surveys
This is a structured way to obtain information from a large number of people.As specific ques- tions are used each time, it is possible to gather quantitative information. It is not normally a useful way for finding out about people’s concerns and perceptions, as it is not possible to ask questions to check on the meaning of responses.
As the questions cannot be changed once the survey starts, it may be necessary to carry out some pre-research (perhaps using secondary material and group interviews) to identify the questions to ask. If the survey is being used to collect a large amount of information, it may be appropriate to draw on professional assistance. For example, a university can help to design the
: ’ ' ' ' ' ' 1 1 ! ' ' 1 ' ' 1 1 1 ' 1 ' ! ' ; ‘ ' ‘ 1 1 ' ! 1 1 1 1 1 1 ' 1 1 1 1 1 1 ! 1 1 ! 1 ' ' 1 1 ' ' 1 1 1 1 1 i 1 1 1 1 1 1 v b 1 1
questions and analyse them properly. 1
‘
a
Monitoring and evaluation: tools for biodiversity conservation and development projects
It is important to identify people to be targeted by the questionnaire. Records, such as course participant lists, can be used to conduct follow-up surveys on people who have attended semi- nars and training events. If you want to get information from the wider population your project is targeting, you will have to identify the best place to locate respondents. In a survey on child health, your survey might get more informative responses if you interviewed women in the market rather than at the clinic.
Surveys are especially useful in monitoring awareness and adoption of project outcomes.
Forms
Forms allow project participants and beneficiaries to collect information on pre-selected topics. The contents of the form are based on selected indicators.At regular periods, the forms are collected and analysed.
This can be a very participatory monitoring method, as it allows stakeholders to play a role
in designing the forms, collecting the information and analysing the forms. The information in the forms can be supplemented by individual or group interviews to find out more about why changes are taking place.
It is important to select a relevant time-frame for the forms, and to collect them at the agreed intervals. Forms can be used on a daily, weekly or monthly basis. It may not be appropriate for respondents to complete the forms on an ongoing basis. Rather, it may be more suitable for monitoring purposes to use forms during agreed periods.
Diaries and logbooks
These documents allow participants to record events over time. They can be kept by individuals, groups and by the project organisation. They can record events and facts, as well as responses and opinions about what happened.
They are flexible tools that can be used in a focused way (such as recording information about farming methods) or in a general way to explore themes. They can provide detailed and qualita- tive information for monitoring purposes.
Participants who keep diaries and logs can come together periodically to analyse the informa- tion they contain, and to discuss and analyse any significant changes. The information can be used for comparative purposes, especially if the same participants maintain diaries throughout the project period.
Photographs and video
A range of techniques can be used, ranging from digital cameras to video tapes, GIS (Geographic Information Systems, which involve satellites), and aerial photography. The pictures and images should be linked to indicators, and suitable intervals identified for taking pictures. The pictures and any negatives should be stored in a safe place, and clearly labelled to avoid any confusion with pictures taken at a later stage.
By taking pictures at regular intervals, changes can be identified and used to stimulate discus- sions with stakeholders to analyse the changes. The changes can also inform questions used in interviews and surveys. The discussion can also focus on what is not visible in the pictures, such as the presence of women.
Calendars
This method can be used to gather information for distinct periods of time, allowing analysis of changes in key indicators over time.As with diaries and logs, they can be kept by individuals or groups. The information can be placed in the calendar by participants as soon as it becomes available.As with other methods, discussion with participants can be used to analyse the infor- mation in the calendar.
61
Oo Ld
Monitoring and evaluation: tools for biodiversity conservation and development projects
More than one type of information can be collected in a calendar. This can make patterns appar- ent, such as the relationship between household labour needs and seasonal agricultural activities.
Calendars can be maintained over a number of years, although most projects will be more interested in information collected at weekly, monthly and seasonal intervals. Summaries can be prepared for longer-term analysis, such as an impact study.
Daily and weekly routines
A variation of the calendar, this method is used to compare the daily or weekly routines of dif- ferent groups of people and of seasonal changes in routine. It can be a useful way to monitor whether the project is placing heavier demands on certain groups and whether project benefits are fairly distributed.
This method can be carried out periodically, rather than on an ongoing basis. It will help to ensure that the information does not swamp the analysers, and that participants continue to be willing to provide the information to the project.
Mapping
Maps can be used to collect quantitative and qualitative information on the geographic distribu- tion of indicators. Information can be recorded manually on aerial photographs or topocadastral base maps and comments offered by participants can be written on the side of the map.
The method can be repeated periodically to assess whether there are any changes in the indica- tors over time. The same map can be used again, with new information recorded in a different colour, or a new map can be developed and compared with the original. Maps are an appropriate way of gathering information about people’s perceptions of conditions in their community.
Ranking
Ranking provides a way for individuals or groups to prioritise a set of options. It can be a very
participative method for allowing participants to identify the importance of the options in rela- tion to one another. This method can provide quantitative information, which can be supported by qualitative information gathered through conversations and individual and group interviews.
The facilitator might identify the options by analysing interview or survey information. In such a case, ranking can be used to verify the issues and learn about the importance attached to each
from the perspective of participants. The facilitator can also invite the group to identify the op-
tions and then to prioritise them.
Monitoring and evaluation: tools for biodiversity conservation and development projects 63
3.3 MANAGING THE INFORMATION GATHERING SYSTEM
The M&E system has to be managed, particularly where there are a number of people involved in gathering information, so that:
# The quality of information is consistent.
* There is a common benchmark for measurement.
=" Information is captured in a form that is usable for evaluation purposes. = It is stored in a place where it will not be damaged, destroyed or lost.
« There is a system for storing information that makes it possible to retrieve an item at any time.
When the system is being designed, decisions have to be made about:
* Who gathers the information. * How it is captured for monitoring purposes. " Where it is stored.
OL
The value of a learning co-ordinator CO
Paula is the person who is responsible for learning and capacity-building within the project.The learning role she plays feeds directly into planning, reviewing and reporting on the project’s achievements. For her to do this, she is actively involved in gathering and storing information. She has information in files and on the computer. She is also a meticulous note-taker. There are certain qualities that she brings to this job that make her so good at it. She is organised, systematic and thorough. She will, for example, spend hours sifting through the information to extract the ‘nuggets-—those gems of information that are embedded within the stories that people tell and the reports that they write but that are not always immediately obvious. She is also good at asking questions so that she can unearth these gems.
(Urban Sustainable Conservation Management Project)
Exercise: Finding the nuggets
Who, in your project, looks for the ‘nuggets’ in the information gathering process? Is this role recog- nised and valued within the project?
Is there anything you could do to develop this role further? (This may require thinking through the formal links between gathering information, and learning, reviewing, planning and reporting.)
S
The value of a database
External staff all feed information into the database, from a range of sites. It is a custom-designed database, on the computer, that includes biodiversity and social data. Everyone can see it at any time and so there is no need for internal reporting. The database is maintained by head office, there is a GIS technician who checks the spatial data and makes sure they have been entered correctly. The data are then fed through to Scientific Services who produce the quarterly and annual reports.
(Conservation Partnerships Programme)
64 Monitoring and evaluation: tools for biodiversity conservation and development projects Ne ee Nee ata atte i ee el ee
It is likely that the gathering will be done by a number of project participants, some or all of whom may not be based at the project office. It is also likely that at some stage the information will have to be transferred to a central point in the project office, either in a ‘raw’ or ‘processed’ form, to be assembled for evaluation purposes or stored as project records. ‘Processing’ could take the form of entering the information into a computer database or writing it up in a standardised format.A flow chart is a useful tool for depicting the flow of information in a monitoring system.
SOURCE | ¢ SOURCE 4 Q;> REPORT T# DONORS <2, ; aan vei REPORT 10 REPORT TO % TARGET GROUP COLLEAGUES
‘>> REPORT TO BiNEFICIARIES ‘D t
SOURCE 2 SOURCE 3 = me mee mee mee me DATA PROCESSING, ANALYSING AND EVALUATION
Two aspects of managing the information system—dquality and storage—will be discussed in more detail here.
What makes the difference between good-quality and poor-quality information?
Information used for monitoring purposes must provide an accurate representation of what is happening. This does not necessarily mean that it has to be 100% numerically accurate or ‘correct’. What is important, is that:
= The whole picture is represented (e.g. that sampling is an accurate reflection of the whole).
= The potential for bias and ‘yea-saying’ is taken into account in the design of interviews, surveys and questionnaires.
# ‘Interference’ or ‘noise’ is identified and avoided or filtered out of the information as far as pos- sible (e.g. the effect of extraneous factors).
* Information that is going to be used for comparative purposes is comparable (the apples and apples vs apples and pears syndrome).
Monitoring and evaluation: tools for biodiversity conservation and development projects 65
Most information gathering techniques need careful planning and design.
= Where there are a number of people gathering the same type of information, the use of forms is a good way of achieving consistency. This can be done with a survey questionnaire, with care- fully constructed closed questions.
= By comparison, a set of open-ended questions is typically prepared for interviews.
Why is a storage and retrieval system necessary?
Many of the monitoring techniques involve comparisons over time, or looking back at project records to identify changes. The material that you collect throughout the project monitoring proc- ess is valuable. It can be used for:
e Comparison purposes
If you collect similar information at regular intervals, you can compare older material with the latest information. This can highlight changes over time that may not be evident when the infor- mation is considered in isolation.
e Impact studies
The information and data that you generate during the life cycle of the project can be used as a source of secondary material during an impact study. This information highlights shifts over time that can help determine the effectiveness of the project’s outcomes.
e Future needs analyses
This information can provide insight into the groups that have benefited from the project, and can help to identify other groups to be targeted by new project initiatives.
Storage systems have to safeguard the material from damage or destruction resulting from fire or floods. If it is in electronic form, there should be a back-up stored off-site. Most importantly, the information should be retrievable. There should also be a mechanism for tracking the whereabouts of any original hard copies that have been temporarily removed from the storage system.
Exercise: What systems for information management does your project have?
Consider the following questions in relation to your project and create a ‘mind map’ of your project’s storage system and its strengths and weaknesses.
|. In what form is your monitoring information stored?
Electronic files—text, spreadsheet, database, other?
Paper files—minutes and notes, forms, records of laboratory analysis?
Videos, tapes or CDs? Maps? Samples?
Any other form?
66 Monitoring and evaluation: tools for biodiversity conservation and development projects
. Where is it stored?
. Who is the person who knows most about where to find something? If he, she or you went on a six-month sabbatical, would someone else be able to find the information needed for M&E?
. Who has access to the information? . Is itin a form that is ready for M&E use or is further processing or analysis needed? . If there was a fire in your office, would you be able to complete your next annual progress report?
. Do you have any recommendations on better management of your monitoring information?
InjSummany,
In Section 3 we:
* Identified that monitoring information can be outcome- or output-related and classified as primary or secondary, qualitative or quantitative information.
* Introduced a number of methods for gathering monitoring information, drawn from the social sciences and from participatory learning and action traditions.
= Explored some criteria that can be used to choose between methods.
* Described ways of managing material once it has been collected to ensure that it is useful and accessible.
The focus of the M&E system should be on gathering information to enhance learning about change and what makes change happen. Reliable methods of gathering information can have
an important influence on the quality and efficiency with which the information is collected. However, the methods should not become an end in themselves—it would be better to select a more suitable method than to continue collecting information of poor quality.
Care should be given to this step of the M&E process. It plays a critical role in enabling participants in the development process to gather information of high quality that will deepen their understanding of the project situation, allowing them to influence learning, decision- making and accountability.
REFERENCES AND FURTHER READING
FEUERSTEIN, M.-T. 1986. Partners in evaluation: evaluating development and community programmes with participants. Macmillan, London.
GUIJT, |. 1998. Participatory monitoring and impact assessment of sustainable agriculture initiatives: an introduction to key elements. SARL Discussion Paper No. |. International Institute for Environment and Development, London. http://www.iied.org/pubs/pdfs/6 1 391IED.pdf (accessed January 2008).
PRETTY, J., GUIJT, |. THOMPSON, J. & SCOONES, |. 1995. Participatory learning and action: a trainer’s guide. International Institute for Environment and Development, London.
WADSWORTH, Y. 1997. Do it yourself social research, edn 2.Allen & Unwin, St Leonards.
Monitoring and evaluation: tools for biodiversity conservation and development projects 67
Section 4: ANALYSING THE INFORMATION
The process of analysing the information does not necessarily occur at a time and place different from that of collecting the information. It occurs when you focus on two important questions:
# What are the significant differences between what we said we would achieve and what we are actually achieving?
= What does this mean for the future of the project? These questions will help you to make sense of the information that has been collected—to
understand what has and has not worked and the reasons for this. This lays the groundwork for acting on whatever lessons are contained in all the information now at your disposal.
As various stakeholders may be involved in providing, collecting and analysing information, it would be surprising if you do not face differences in and disagreements on how the information is under- stood and interpreted.An important part of this process lies in acknowledging the various inter- pretations, and allowing them to inform your own perceptions and understanding.
In analysing the information, you are trying to make sense of it so that you can make recommenda- tions for appropriate action.
By the end of this section, you will have:
" Guidelines for assessing actual outcomes and outputs against expected outcomes and outputs. =" Guidelines for assessing assumptions.
Explored the value of building shared understanding among different stakeholders and how this can be done.
= An overview of different types of evaluations and evaluation methods. = Been provided with a sample Terms of Reference for contracting external evaluators.
= A deeper understanding of how social development evaluators work.
68 Monitoring and evaluation: tools for biodiversity conservation and development projects
4.1 ESTABLISHING IF THERE ARE SIGNIFICANT DIFFERENCES
This first step in this process is partly descriptive and partly comparative. Its purpose is to identify what was supposed to be happening at this stage of the project (this information will be in your plan) and to compare it with what is actually happening. The questions you will be seeking to an- swer are:
« Are there any differences between what was planned and what was actually achieved? In other words:
» Did the project accomplish more than what was expected?
» Did the project achieve less than what was expected?
As the real world is a dynamic, ever-changing environment, it is highly unlikely that the actual situ- ation in which you find yourself will be identical to the intended situation. It will probably not be helpful to describe all the differences between the actual and intended situations, as some of the changes may be relatively small and of little importance.
However, it will be important that you describe any significant changes. The significant changes are interesting because they will form the focus for your analysis and subsequent action. The changes can be identified by monitoring the indicators at the level of outcomes and outputs, as well as by monitoring the enabling factors or assumptions. (These are components of the project plan. The definitions are provided in Section |.)
There is usually one guiding question for each level of indicator. e At the outcomes level, the guiding question is:
* Is the project having any positive influence on the target group, beneficiaries and/or environ- ment?
This question focuses us on outcomes at the level of Development Goal and Project Purpose. The indicators you have developed for your outcomes will help you assess the degree of change that has taken place. Project monitoring activities are likely to focus on the intermedi- ate targets that you have established, as this will allow you to review the changes in the project environment over the life of the project.
e At the outputs level, the guiding question is:
* Is the project making progress towards its objectives and how efficiently is it using resourc- es?
Here you are looking for information about the project’s progress in implementing its planned Activities and achieving its key Outputs. It is important to pay attention not only to what has been achieved, but also to the quality and the scale of the achievement.
The focus of output indicator monitoring, especially at the start of the project, will be on progress towards implementing the Activities described in the project plan. The Activities are the easiest part of the plan to change in response to problems and unexpected events. By monitoring the progress of the Activities, you are able to respond quickly to emerging issues before they escalate into larger problems.
Monitoring and evaluation: tools for biodiversity conservation and development projects 69
You can develop a false sense of security in the status of the project if you simply monitor its progress towards its goals. There are many projects that have made a good start with progress towards their results, but have run out of resources before the end.As a consequence they have struggled to complete all their planned activities. By paying attention to issues of resource use and efficiency from the start of the project, you will help to ensure that your project will have sufficient resources to achieve all its objectives.
The two main resources over which the project has some measure of control are staff time and finances. Both of these can be viewed as ‘consumables’-—once they have been used for one thing, they cannot be used for something else.
Staff make efficient use of their time when: » They use as much time as was scheduled to complete planned activities. » There is an appropriate number of people working to complete a task.
» They are doing work appropriate to their skills and experience.
Efficient use is made of financial resources when: » The budgeted resources are used to complete a task. » There are few quality problems that require tasks to be repeated.
» Appropriate savings are made on unnecessary expenses.
Members of the project team should aim to monitor and control costs as they carry out their work. This is more likely to happen if they are clear about the resources that are available to them. By developing an activity-based budget for the project, and providing regular (monthly) financial reports, it is easier for staff to monitor and control expenses.
Establishing the nature of the difference
Are there significant differences between what we said we would achieve and what we are actually achieving?
The monitoring information may show that the expected changes are taking place. If so, this affirms that the project intervention is appropriate in bringing about the desired changes.
However, if you find that there is a significant difference between what you expected and what is actually taking place, the situation requires further investigation. It is under these circumstances that the views of an external evaluator can be particularly helpful. The aim of this stage of analysis is to deepen your understanding of the situation and to inform the recommendations you might make for responding to this unexpected development. This is achieved by asking the questions:
= Did we achieve more than we expected?
= Did we achieve less than we expected?
Ss
70
Monitoring and evaluation: tools for biodiversity conservation and development projects
Did we achieve more than we expected?
Significant changes in what you find will not always be for the worse. There might be occasions when you find that there is a greater response than you expected, or a positive outcome that was not foreseen at the planning stage.
In the original plan, we were going to appoint negotiators. But we found that landowners were willing to negotiate with conservation staff, and so we decided not to appoint negotiators, but to use extension staff instead.
There is also much skills development going on but we are not consciously monitoring it.
(Conservation Partnerships Programme)
It might be important to understand why this has happened, as it may help you to identify how you can use this greater response to the project’s benefit. It may also help you to have a better understanding of the conditions under which you are working.
Some questions that might guide your investigation into achieving more than you expected at the outcomes level are:
* Is the increased response a result of the project’s work, or is some other factor responsi- ble?
* Is the response likely to place greater demand on the project’s services in the next period of implementation?
= Are the project’s targets sufficiently ambitious? Should the objectives be stretched further to reflect this new reality?
When investigating higher-than-expected achievements at the outputs level, you are looking for information on why it has happened, and whether there are any future implications for the project:
# What is driving the increased demand?
= Can the project accommodate it?
" If so, can you adjust budgets and plans to meet the raised targets?
= If not, can you work with other organisations to pick up the surplus demand?
= Can you reduce expectations in any way?
You may find that you can complete tasks in less time than you anticipated, and that there are less expensive ways to obtain certain services from various suppliers. When you make faster progress or better use of resources than you expected, you have the opportunity to re-allocate
either the time or resources to other activities—as long as you are monitoring the situation and become aware of it early enough.
Did we achieve less than we expected?
If monitoring reveals that the rate of implementation is slow, with few benefits materialising, it may suggest that the project is not following an intervention strategy appropriate to the context.
Monitoring and evaluation: tools for biodiversity conservation and development projects 7\
One of the purpose-level indicators for the pilot project was the adoption of contractual conservation agreements by landowners. By the end of the pilot, no contracts had actually been signed, although several were pending. We had not fully understood the complexity of land ownership.
(Conservation Partnerships Programme)
Further investigation will be needed to understand why the project is not progressing as planned.
At the level of outcomes, the investigation may reveal that:
# While the intervention strategy may have been appropriate when the project was first planned, changed conditions mean that it is no longer so.
« The project was planned with insufficient information and a limited understanding of the context, resulting in an inappropriate or over-ambitious intervention.
= There might be problems with the quality and timing of the project’s work.
If outputs monitoring shows that there is a slower rate of implementation than expected,
that there are significant delays in starting or completing certain activities, then you are facing progress problems. There could be many reasons, ranging from staff issues to external factors. These problems with progress can represent a significant threat to the success of the project and so it is important to identify them at an early stage when it may still be possible to prepare a response that can be quickly and easily implemented.
Over-spending or poor use of resources can be a rich source of learning for the project:
= Costs may have risen since the budget was prepared. This can be a timely warning for revi- sion of the budget.
= Over-spending may be due to poor financial management, such as taking on additional activities. The budgets might have to be adjusted to make up for the additional expenses by reducing the finances available to other activities.
Two useful resources on financial management include: Shapiro (1995, Financial management for self- reliance) and Randel (2001, Planning for implementation).
At the assumptions level, the guiding question is:
= Are the enabling factors present or are they absent?
By monitoring the enabling factors that will contribute to project success, you are not looking for progress or reaction. You are assessing the presence, or the absence, of these critical enabling fac-
tors.
The external factors, by definition, are beyond the control of the project. The reason for including the enabling factors in the monitoring process is to provide an ‘early warning system’ that can draw the attention of the project team to potential risks to project success.
72 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Are the enabling factors present?
Enabling factors are also known as assumptions. By definition, their potential influence on the objectives is unknown when the project is planned. The assumption of the planners is that the enabling factors will be present and will positively support the implementation of the project.
Understanding and support for the stewardship concept has grown remarkably—much faster and more widespread than we had anticipated. The willingness of landowners to negotiate with conservation staff was underestimated.
(Conservation Partnerships Programme)
As you gather information about the enabling factors, you may find that the assumption has proved true. In the example above, the project developer is referring to the assumption that the project would receive the support of the target group. This is important feedback for the project team. It means that the project strategy continues to be valid, as one of the critical enabling factors is supporting the project’s work.As there is little significant difference between what was expected and what the monitoring information is revealing, the project does not have to respond in any way.
If your enabling factors are in place as expected, you will probably have to continue to moni- tor them at regular intervals for the rest of the project period. The external world is a dynamic place, and a change in the status of the enabling factors could have many causes.
The strong appearance of an enabling factor that significantly exceeds planned assumptions is not necessarily a good development for the project. The project team may be tempted to aban- don the project plan and to revise the strategy, taking the new situation into account. While conditions may be supportive at present, it may not be possible to predict how long this will continue, and whether it is sustainable.
Great care should be taken when deciding whether or not to increase the ambition of the project strategy and objectives because of such changes. There may be times when it will be beneficial to stretch the targets set for the objectives in response to a positive development. But there may be other times when it might be better to maintain the project objectives at their original level of ambition.
e Are the enabling factors absent?
As the enabling factors are outside the control of the project, it is not possible to know when they will come and when they might depart. Ongoing monitoring of the enabling factors can help to identify the time when a factor changes its presence, and thus its influence on the project.
There are a number of conflicting and confusing policies that are coming out of the different national departments. This has made it difficult to secure water-tight legal agreements with the landowners.
(Conservation Partnerships Programme)
Monitoring and evaluation: tools for biodiversity conservation and development projects 73
By waiting to see what the longer-term trends will be, and whether there are short-term fluc- tuations in the status of the enabling factors, the project can identify the way it should respond. Before it decides to make any changes in the project plan, it might be worth investigating the forces that may be influencing the enabling factors. If these forces and systems can be positively influenced, the enabling factors may become more consistent, to the project’s benefit.
Should the enabling factors continue to fluctuate, or be absent, the project’s success is threat- ened. It will be appropriate to investigate the situation more closely and to start a problem- solving process to protect the work that has gone into the project.
Exercise: A snapshot view of your project's progress
Taking into account the stage of implementation of your project, consider the following questions:
éS
In which areas of delivery has the project achieved more than it said it would? Where has it achieved less than expected? What are some of the reasons for the difference?
Which assumptions are you monitoring? Have there been any changes in these external conditions that are critical to the success of the project? Are there any assumptions that should be changed, added or dropped?
74 Monitoring and evaluation: tools for biodiversity conservation and development projects 4.2 CLARIFYING AND AGREEING ON THE REASONS FOR ANY DIFFERENCES
The purpose of M&E is to help you review your work in progress and to see whether or not you have to make any changes to the way you are implementing the plan. If you find that there are some significant differences between what has taken place and what was planned, you are likely to want to respond to them in some way.
Before you act, though, it is important to be clear on what you are responding to. While it is some- times possible to spot the reasons for why your plan did not proceed as intended, the members of your project team and other stakeholders may hold different opinions on these underlying causes.
This step is important, as it will help you to clarify the reasons for the differences and also to build agreement among important stakeholders about these reasons. This is an important foundation for what will follow—acting on this new insight and understanding.
Differences in outcomes and outputs can be ascribed to external or internal factors. It is helpful
to make this distinction as a starting point. Enabling factors are almost always external. It is also useful to probe beyond what appears to be the ‘obvious’ reason for a mismatch between what was planned and what actually happened in case there are deeper underlying causes. The picture that
is being conveyed by the M&E information may be being influenced by, for example, a bias in the selection of indicators, which would start to become apparent at this stage through a process of interrogation and clarification.
Some of the tools that may assist you in clarifying and reaching consensus on the reasons for differ- ences, and what this might mean for the project, are discussed here.
One of the challenges in analysing the information that you are collecting on the project’s out- comes and outputs, is to build a shared understanding among the different stakeholders about the meaning of this.
If stakeholders in the project do not agree with one another on the causes for why things turned out differently from what was expected, it will be difficult to get support for any changes that you may propose to the way the project is carried out.
Shared understanding forms the basis for joint ownership of the analysis and for getting support for any changes in the project’s approach.
Monitoring and evaluation: tools for biodiversity conservation and development projects 7S
The process of building a shared understanding should not try to suppress the different opinions that people may hold because of their differing experiences and mental models. Rather, a dialogue on what
is happening, and why, allows participants to share their perspectives, knowing that their colleagues will listen to them.
In such a dialogue: = People are expected to challenge one another and the team’s mental models (more information on this is presented in the next section). " Shared understanding will emerge as the team makes sense of the conflicting perspectives people hold. = Consensus may not result, but people will have a better appreciation for the reasoning others are using, which can lead to an appreciation for the recommendations for changes that will be made.
Participants in the dialogue can range from two people to the whole team, and it can include stakehold- ers from outside the project, such as target group representatives.
Reference: Raelin (2000, Work-based learning)
As we review the monitoring information, we may find that we did not meet the desired results be- cause of the way in which we carried out the work. There may have been problems with the quality of our work. Perhaps our planning and assumptions about what was needed were incorrect from the start.
Often, there can be pressure to overlook these causes and to look for other reasons for why things did not go as planned. It can be uncomfortable to find the fault lies in the way the work was carried out or managed, especially if senior people in the organisation were involved.
The pressure to pass over poor management and capacity problems can be stronger in organisations that receive donor funding. If staff are concerned that the organisation will not receive further funding because of its poor performance, they may be reluctant to reveal the underlying causes of the prob- lems.As a result, the project and the organisation are less likely to learn from these experiences and to develop ways to improve their ways of working.
For staff to identify and discuss openly any internal, organisational problems that affect the project's re- sults, the project team should support problem-solving and reflective discussion. In such a work climate:
=" Team members are trusted and supported by their colleagues. = Staff are encouraged to take responsibility for the consequences of their work. = Problems and mistakes serve as a source for reflective learning, not for blame.
»))
When we had our last evaluation, we decided that we would be honest and open in sharing the findings za with others. Obviously, there were things that were and were not working about the project.We dared «! to say that things were not perfect, that we had got some things wrong. We shared this with our Project
Advisory Group and with the C.A.PE. Implementation Committee. | think this opened people up to the possibilities of what an evaluation can do. Our donors were also very supportive and impressed. So were
the environmental educators. | think people were able to learn through our experience.
i i] | | a a (Urban Sustainable Conservation Management Project) 1 if i (Senge 1994) .
t
76 Monitoring and evaluation: tools for biodiversity conservation and development projects
4.3 CONDUCTING FORMAL EVALUATIONS
Formal evaluations have a useful role to play in analysing information about the progress of the project. Much of this evaluation work can be done by internal project staff. However, another way of bringing new insights and perspectives to these questions is to make use of external evaluators. These evaluators can bring certain skills and expertise that are not necessarily located within the project.
There are different types of evaluations, as well as different evaluation methods, that can be drawn on. The selection of the most appropriate type and method will depend on the project particulars: the stage in the project life cycle and the type of activities being monitored.
Types of evaluations
Specific types of evaluation have been developed to address the various evaluation questions that are relevant at different points in the project cycle.
e Ex-ante
An assessment of the project plan and approach, before it starts, to assess whether or not it has a good chance of achieving its objectives.
e Midterm
A review, carried out midway through the life of the project to determine if the project is likely to be successful and to assess whether or not any changes in approach should be made to ensure SUCCESS.
e Terminal An evaluation at the close of the project to determine if it has accomplished its objectives. e Impact
An assessment of whether or not any benefits resulting from the project have led to a sustained improvement in the issues it was addressing.
One final distinction in monitoring and evaluation terminology should be noted.
e Formative evaluation
Evaluation that is ongoing, or that provides continuous feedback to inform ongoing changes and improvements in a service or a product.
e Summative evaluation
Evaluation that is done periodically to inform reporting and decision-making. The decisions are typically at a high level, such as whether or not to continue funding for a project.
In almost every situation, monitoring activities provide formative evaluation of the project’s work. Cer- tain evaluation activities, such as a midterm evaluation, are also formative, as they provide feedback that is used to make changes in the project’s work. Other evaluation activities, such as the impact evaluation, are summative in purpose, as they are used to inform decisions on whether or not to continue with the project.
Both formative and summative evaluations allow us to ask, ‘What can we learn from this experience that will help us tomorrow, or the next time, or may be relevant in other places?’ Formative evaluations answer this question in ways that directly inform project implementation, while summative evaluations provide insights to those further away from the work, such as donors and host programmes.
Monitoring and evaluation: tools for biodiversity conservation and development projects 77
Particular methods that are useful for evaluation purposes include:
e ‘Before and after’ studies
These studies are based on baseline information. Subsequent assessment compares the current status with the status at the starting point.
e ‘With and without’ studies
Assessments are made of both people who adopt and those who have not adopted the project’s services. The study aims to assess whether or not the project is making any positive difference among those who are adopters. The nonadopters can provide information on their awareness of the project’s services, and why they have not adopted them.
e Longitudinal studies
An ongoing assessment is made with individuals or groups, selected as representative of the wider target and beneficiary population. Changes in this sample group are assessed periodically. The group can include both adopters and nonadopters.
The process of contracting the evaluators has to be well thought through and rigorous—both in terms of the legal requirements and the objectives of the evaluation. Clear Terms of Reference (TOR) are an important starting place to ensure that you get the most out of this process.
Furthermore, different evaluations call for different ways of working. Status assessments, for exam- ple, require a specific set of skills and expertise, as do ElAs. One of the challenges facing projects in the biodiversity conservation sector, is how to assess the progress and impact of their social objectives. Social development evaluators bring a different approach and set of skills to their work. The table that follows the sample TOR outlines some of the principles and practices that typically inform the work of social development practitioners.
78 Monitoring and evaluation: tools for biodiversity conservation and development projects
SAMPLE TERMS OF REFERENCE FOR EXTERNAL EVALUATORS (FROM CAPE FLATS NATURE)
CFN began in June 2002 as a partnership between the
City of Cape Town, the South African National Biodiversity
Institute, the Table Mountain Fund (WWF-SA) and the Botanical Society of South Africa. Cape Nature and the Table Mountain National Park joined the partnership in 2006.The project’s purpose is to build good practice in sustainable management of City nature conservation sites in a way that benefits surrounding communities, particularly those where incomes are low and living conditions poor.
CFN was initiated with seed funding from the City and the Table Mountain Fund (TMF), as set out in a three-year contract between the City and TMF on the one hand, and the South African National Biodiversity
Institute (SANBI) on the other hand as the implementing
partner. The completion of this contract at the end of
June 2005 formally brought to an end the demonstration
phase of the project, and called for a major evaluation. The evaluation was conducted by an independent external evaluator.
As part of the demonstration phase, CFN raised additional funds from the Critical Ecosystem Partnership Fund (CEPF).This required the drafting of a logical
framework, which added further content to the City/TMF
contract, and also extended the terms of reference of the project to design a replication and roll-out strategy. The 2005 evaluation constituted a mid-term review
for the CEPF-funded project, and its recommendations contributed to the development of the replication and roll-out strategy. The CEPF funding comes to an end in September 2007, and the contract with CEPF requires a further evaluation to be completed.
CFN again requires the services of an independent
external evaluator to facilitate, conduct and document this evaluation.
Objectives The objectives of the evaluation are as follows:
= Comprehensively document developments in the his- tory, strategic approach and organisational processes of the project since the previous evaluation.
= Assess the extent to which the project has met the
objectives of its contract with the CEPF, paying particu-
lar attention to the sustainability (including financial sustainability) aspects of the project.
" Assess the relevance of the outputs to the purpose of the project as set out in the CEPF logical framework.
# Strengthen the project by providing external perspec- tives, insights and, where appropriate, recommenda- tions based on broad developmental experience.
" Develop a draft framework for phased levels of sup- port in establishing new sites, and for building and maintaining civil society support for conservation, particularly in low-income communities.
= Develop an outline and a process for the development of a handbook on integrated urban nature conserva- tion management to be completed by 2009.
= Draw and document lessons learnt from the develop- ment of the replication and roll-out strategy in a way that contributes to the implementation of this strategy.
" Build the capacity of the City’s Nature Conservation Branch to write up case studies of their work in a way that contributes to spreading a people-centred nature conservation practice by documenting and sharing lessons into the future.
" Develop the ability of Cape Flats Nature’s Capacity Building Manager to build capacity of other partners, e.g. Cape Nature to write up case studies of their work in a way that contributes to spreading a people- centred nature conservation practice by documenting and sharing lessons into the future.
Method
The method used should facilitate organisational learning, at the same time enabling accountability to funders. It should integrate the evaluation process into the organisational life of the project and that of the City Nature Conservation Branch as far as possible to avoid creating additional work for the project team and its key partner, and to ensure support for the implementation of recommendations that emerge.
An effort should be made to use both qualitative and quantitative methods of assessment as far as possible, and recommendations for future information gathering will be made where the necessary information is not available.
T
Monitoring and evaluation: tools for biodiversity conservation and development projects 79
The Project Advisory Group and project team should be involved in finalising the contract and evaluation process with the external evaluator.
CFN will arrange meetings for the process, e.g. the Project Advisory Group and Champions Forum, but the consultant will set up individual interviews.
Outputs
" A City Nature Conservation Branch capacity-building workshop on writing up case studies.
A report that synthesises all materials and the outcome of interviews, and develops an independent perspective and interpretation. It should conclude with concrete recommendations covering all areas of the TOR.
A draft framework for phased levels of support in establishing new sites, and for building and maintain- ing civil society support for conservation, particularly in low-income communities.
An outline and a process for the development of a handbook on integrated urban nature conservation management.
Reporting and management
CFN’s Project Manager will manage the implementation of the evaluation, and the consultant will report to her.
Timeframes
Proposed timeframes, to be finalised in consultation with the evaluators and the City Nature Conservation Branch (for the case study workshop), are as follows:
® The finalisation of the consultant contract and evalu- ation process will take place in time for a Project Advisory Group meeting in May, to form part of the process.
" The case study workshop will take place in May, with case studies due in draft by early June, and finalised by the end of June.
= The Champions’ Forum (drawing together community champions from the four pilot sites to share experi- ences and draw lessons) on 2! June may play a role in the evaluation.
= A draft report will be submitted by the end of July for comment, and the report will be finalised by the end of August.
Reference documents
Project materials to be consulted in framing the evaluation include:
= CEPF logical framework.
= City/TMF contracts.
= Reports to funders and funder comments on these reports: biannual narrative reports to CEPF and Phase 2 reports to City.
= Soal, S. & Van Blerk, R. 2005. Report to Cape Flats Nature on the outcome of an evaluation.
Roll-out strategy outputs.
Financial strategy.
# Memorandum of Understanding.
Hill, S. 2006. Networking people and nature in the city.
= Videos: Cape Flats Nature: biodiversity for the people; Bridging the gap: new conservators speak; Networking people and nature in the city.
Case study file.
Press clips file.
80 Monitoring and evaluation: tools for biodiversity conservation and development projects
Principles underpinning a social development approach to evaluation (extract- ed from CDRA's website)
e Learning and accountability
External evaluation should enable both learning and accountability. Learning is to search out understanding, insight, knowledge or skill through reflecting on experience. Accountability is a transparent story of one’s successes and failures. When working with small-scale social change (where work occurs in the realm of the invisible, personal and interpersonal), organisational learn- ing is the most reliable route into external accountability. In CDRA, we have discovered that when we share our own field experiences for the purpose of learning and improving future practice, ac- countability happens. It flows naturally. But we also know that when the spotlight is on ‘Account- ability’ (with a capital ‘A’), the liberty to be vulnerable shrivels, and the need to prove success
or failure dominates. The outcome? Superficial learning and accountability; a neat report for the donor;a tick in the manager’s to-do column.
Quantitative and qualitative measures
In this approach to evaluation, quantitative and qualitative measures are combined. Ideally, if or- ganisations have mechanisms for feedback and data collection, basic quantitative information on ‘outputs’ is available as a starting point for the evaluation. If not, the evaluation has to begin with generating some of the required information, and if time allows, recommend systems for future use. However, where feedback on ‘quality’ and strategic appropriateness is sought and where different strategies are valued in different ways, quantitative measures are insufficient. Qualitative measures and an approach that draws on the understanding of all involved are far more effective in providing a meaningful indication of impact and value.
Measuring impact—evaluation or research?
Measuring the impact of developmental interventions is critical to the process of learning and ac- counting. It is possible and necessary to collaboratively measure impact, quantitatively and quali- tatively, on those directly involved in and directly impacted by the intervention, and from which one can strategically argue or anticipate the likely contribution of that intervention to change downstream'. But measuring or proving impact downstream, scientifically for governments, on the ultimate ‘beneficiaries’ and wider, is considerably more complex and not a task that practitioners can be expected to take on as part of their direct evaluations in practice.
Practitioners may be able to demonstrate and illustrate impact downstream through following up particular case studies, but scientifically measuring impact downstream requires developmen- tal research, requiring skills, resources and time that they do not normally have. Donors have to fund research for this impact on ultimate and wider ‘beneficiaries’, relieving the pressure on M&E systems, but contributing the research to their learning and enabling practitioners to focus on evaluation for practice development.
Participation
Essentially, evaluation for learning does not substitute for facts and figures—but goes beyond it. It is quite literally ‘formative’ as participants’ understanding of their work or experience is deep- ened—formed—through the process of evaluation.An evaluation ‘finding’ is arrived at through a form of structured conversation, a process of questioning, listening, making meaning and deepen- ing understanding. This kind of ‘finding’ is both a valuable and valid one. It builds on the logic of ‘doing is knowing’, and validates the knowledge generated by considered experience.
' Outcomes mapping
Monitoring and evaluation: tools for biodiversity conservation and development projects
The central and most important source of information in developmental external evaluation is people themselves. Their perceptions (be they reflective of a majority viewpoint, or not) are valid material for the evaluation. The processes being evaluated are human processes, therefore the measures must be human too. Seen in this way, a ‘perception’ of impact is a form of impact in itself.
The external evaluator’s role is to create an environment that is conducive to real engagement. Engagement in self-reflection, group exchanges and in one-on-one interviews. It is his/her role to help participants listen, to themselves and others, as both a form of accepting and of asking why. It is his/her role to help participants to view their scenarios from different perspectives, and to spark their dynamic for change. The evaluator is required to sink deeply into each perspective be- ing offered, yet retain a critical and independent stance. This is what frees her/him to incorporate all contributions towards providing a whole picture of the organisation’s functioning and impact.
Internal and external foci
Exploration for the reasons behind things working, or not working, requires bringing the internal organisational aspects into play with the external factors.
The internal aspect of evaluation is helpful because it addresses those things that can be changed or improved upon. It frames both problems and solutions in such a way that they are within the reach of the organisation.
An evaluation addressing only the external might conclude that an organisation failed simply be- cause of a shift in the external environment. However, viewed from the inside out, the evaluation could conclude very differently: perhaps that the organisation did not have adequate environmen- tal scanning systems...or that it never reviewed its strategies...or that its leaders were not fulfilling their duties...or that there was poor team work.
You can do something about setting up effective scanning systems. You can do something about
that struggling leader. But you cannot control the outside world. In other words, if you make space
to consider the internal questions alongside the external ones, the conclusions will be different and they will hold insights that are organisationally do-able.
Strategic connection
Finally, evaluations should be conducted with an eye on the future. If you want the evaluation to
be of real use to your organisation, the implications of all the findings and potential solutions must
be teased out. So, team work is poor. How might things be done better? What has to change? What would that mean for individuals and for the organisation?
Final recommendations should incorporate the outcome of this reflection and learning and specify
the subsequent ‘ideal’ goals (strategic and organisational), as well as the processes and paths to be followed to begin reaching those ideals.
(Soal et al. 2007)
8|
82 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Drawing up a terms of reference document
Taking into account the different types of evaluations, is it time for you to conduct a formal evalua- tion of the project?
If yes, consider the following questions:
What is the purpose of the evaluation?
What level of objectives are being evaluated? What are the intended outputs of the evaluation? Who will manage the evaluation process?
Will it be an internal or external evaluation? Or a combination of both?
Draw up a draft terms of reference document.
In Summary,
In Section 4 we:
* Looked at the process of comparing actual achievements against planned achievements and how this applies to outcomes, outputs and enabling factors.
* Introduced tools to assist the project team and stakeholders to find common ground on the reasons for significant differences between what was planned and what monitoring reveals as having been achieved.
= Explored the value of formal evaluations and, in particular, how social development practi- tioners would approach an evaluation.
This stage of the M&E process is critical, as it forms the bridge between reflection on the events and experiences of the past, and future action. It does this by investigating the signifi- cance of what has happened.
Section 5 takes this one step further by looking at the way to identify recommendations and the decision-making process for acting on the analysis.
REFERENCES AND FURTHER READING
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ. RANDEL, M. 2002. Planning for monitoring and evaluation. Olive Publications, Durban. SENGE, P. 1994. The fifth discipline fieldbook. Nicholas Brealey, London.
SHAPIRO, J. 1995. Financial management for self-reliance: a manual on managing the finances of a non-profit organisation. Olive Publications, Durban.
SOAL, S., HILL, S. & REELER, D. 2007. Evaluation: a developmental approach. http://www.cdra.org.za (accessed January 2008).
Monitoring and evaluation: tools for biodiversity conservation and development projects 83
Section 5: ACTING ON THE ANALYSIS
There are two factors that are important for taking action once you have a good analysis of the information you have collected: drawing out recommendations and effective decision-making. The value of a formal evaluation (whether it is conducted internally or with the support of external consultants) is that it helps to create the time and space for these ideas to be identified and fur- ther explored. There are some broad categories into which recommendations typically fall and which can be used to think through the future of the project, even in the absence of a formal evalu- ation process. They are discussed below.
The likelihood of new actions being adopted and real changes being introduced into the project relies on effective decision-making. In particular, it is important to consider who should be part of the decision-making process, and what is being decided.
By the end of this section, you will have:
# A framework for identifying different types of recommendations. = Explored different components of the decision-making process. = Looked at how decision-making occurs in groups.
fl 1 1 i : i i 1 1 ! ® Been introduced to the notion of ‘resistance’. : a
Monitoring and evaluation: tools for biodiversity conservation and development projects
9.1 DRAWING OUT RECOMMENDATIONS
In this step, we start to shift our focus from past experiences towards future actions. This involves responding to any significant changes in the project and its environment by:
* Developing specific recommendations for follow-up action. by project staff.
* Identifying general lessons from this experience that might be relevant in other contexts. Applying insights from one part of the project to other parts of your work can help you to avoid encountering the same problems in the future. This can also help you to respond to the changing
context in which you now find yourself, allowing changes to the project plans that take account of your environment.
Now that you are aware of any significant changes, and understand why they have occurred, this part of the project monitoring process allows you to ask:
= Are there any significant consequences for your project?
* Should you do anything differently in future?
* What specific recommendations do you have for decision-makers?
Any recommendations you develop should be viewed as experiments. You do not know if the proposed ‘fix’ will achieve the desired result and what will happen. But by adopting an experimen- tal attitude as you consider possible responses and develop your recommendations, rather than ‘we must get this right’, you encourage innovation and risk-taking. This can help you to identify
responses that you might not normally have considered, allowing you to accomplish more than you had considered possible.
identifying recommendations
Now that you have a better idea of what is happening, and why, you may want to respond in an ap- propriate way. Your response may help keep the project on track towards its objectives, or it may bring about a change in the project’s objectives.
By drawing on the lessons you have identified, you may identify practical steps you can take in re- sponse to the changing environment. Should you:
= Keep doing?
» Do more of or less of...
» Do in different locations... = Stop doing?
= Start doing?
What kinds of changes can you recommend?
Recommendations for change can be viewed as a process of ‘steering’ the project around any iden- tified obstacles that might prevent it from reaching its destination. If any changes are needed, they should be made in proportion to the obstacles you are trying to avoid.
Monitoring and evaluation: tools for biodiversity conservation and development projects 85
One helpful way to decide on steering changes, is to consider the following four options:
e No intervention
No response will be necessary when there is no significant change between the planned and the actual status of the project.
Three, instead of four, consultative meetings were held with the landowners. They are in agreement with the = proposed plan of action regarding the removal of alien vegetation on their land. ~/)
(Source: Project Developers’ Forum, |8—19 March 2008)
e Corrective action
This is appropriate when there is a difference between the planned and the actual status that can be corrected relatively easily by:
= Adjusting or repeating certain activities. or
# Re-allocating inputs (such as financial resources and staff).
The project was unable to secure meetings with the community representatives in the time allocated. The Zz remoteness of these communities and the bad roads contributed to this. Meetings have been re-scheduled for \G the next quarter.
(Source: Project Developers’ Forum, |8—19 March 2008)
e Re-plan or adjust the plans
This change can be introduced when there is a significant deviation from the planned status that cannot be corrected immediately. Responding to the deviation may require making changes in the project plan at the levels of:
= Development Goal or Project Purpose objectives. = Outputs and Activities.
= Inputs.
One of the communities with whom the project is working is tied up in a complicated land claim. Instead of asking community members to dedicate all their time to maintenance of hiking trails, the project has re-allocated some of this time to supporting community reps in understanding their rights and the legal processes they have to follow.
(Source: Project Developers’ Forum, |8—19 March 2008)
e Discontinue the project or individual project components
This extreme action may be necessary when the monitoring shows that the project’s objectives cannot be achieved with the current intervention strategy and resources.
86 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Steering the project—different options
Look back at the various phases of the project in which you are involved. Can you find examples from your own experience of the four categories of recommendations?
No intervention.
Corrective action.
Re-plan or adjust the plans.
Discontinue the project or individual project components.
What were your reasons (or thinking) for taking these different courses of action? Knowing what you know now, would you take a different course of action?
Monitoring and evaluation: tools for biodiversity conservation and development projects 87
J.2 EFFECTIVE DECISION-MAKING
A key purpose of learning is to inform effective decision-making by providing timely and relevant analysis. The decisions that are made on the basis of this analysis can support adaptive management by introducing necessary changes to the project plan and to the supporting budget.
The decision-making process builds on the recommendations that have been developed, respond- ing to significant changes identified during the monitoring process. This will result in a clear justifi- cation for any proposed changes to the plan and budget.
It is important to clarify who will be involved in decisions about changes because: = Representation of important stakeholders increases ownership and support for decisions. = It will allow different views to be considered during the decision-making process.
= It can reduce delays by assigning certain decisions to staff more closely involved in project im- plementation.
If an Operations Manual or Policy Document is developed at the start of the project, it can include a section on decision-making responsibilities and processes. Such a document should be reviewed occasionally to keep it up to date with any changes that have taken place in the organisation.
The manual or policy should identify the decisions to be made by: » Staff. » Management.
» The board or steering group of the project.
In consultation with:
» The target group, beneficiaries, donors and other stakeholders.
The principle of subsidiarity should be kept in mind when deciding how decisions will be made. According to this principle, solutions to problems should be identified and acted on as close to the source of the problem as possible.
If you act on the basis of this principle, each succeeding level of management will only deal with the problems that cannot be resolved at a lower level. This can relieve management from a great burden of having to approve all changes. It also means that management can focus on dealing with blockages in the system that cannot be addressed at a lower level.
This principle also supports the view that the focus of M&E is on providing project staff with timely information for implementing the plan. If all decisions have to be passed through the project’s man- agement hierarchy, the staff may be blocked from developing appropriate and timely responses to emerging problems.
Monitoring and evaluation: tools for biodiversity conservation and development projects
How will decisions be made?
Agreement on who will take various decisions will not, by itself, result in healthy decision-making processes. It is also important to pay attention to the way the decisions will be made.
It is useful to keep in mind that organisational culture has an important influence on M&E.An or- ganisational culture that is open to feedback, curious and respectful of others, will support timely decision-making that includes the views of interested stakeholders. The participation of these stakeholders can end up having little or no influence if a small group ultimately takes decisions in an autocratic manner. There may be times when decisions have to be taken quickly, without much time for consultation with affected stakeholders. There may be other occasions when legal or con- tractual requirements direct the process, overriding the views of certain stakeholders.
There are a number of methods that a group can use to make decisions. Some of these options are identified on the following pages.
What is being decided?
Decision-makers are being asked to approve a recommended change. They may discuss whether or not they think that the recommendation is appropriate to the change that it is responding to. They may want to look at the information that has been used to make the analysis. If major changes are being recommended, the decision-makers may want to carry out a diagnostic study to be sure that that this is an appropriate response.
Decision-makers may want to adapt the recommendation. It would be helpful to consult with the people who made the analysis to be sure that they are not overlooking some important reasons for the initial recommendations.
When the decision to approve a recommendation is made, it will be necessary to ensure that it is properly embedded in the project by identifying:
e Whowill carry it out?
Assign responsibility for implementing any changes. e By when should it be implemented?
Clarify how much time the implementers will have to make the changes. e How will the changes be monitored?
Agree on the indicators that will be used to check whether or not the recommendation has been implemented, and whether it is having the desired influence on the project and its context.
If staff are to implement the decisions in the ways intended, they need to understand the reasoning behind the decisions. This is more likely to happen if staff are also involved in the process of col- lecting and analysing information. No matter to what extent staff have been involved, communica- tion still plays an important role in ensuring that people are clear about why decisions have been made, and what is expected of them. Staff may be reluctant, for a range of reasons, to carry out the decisions. It will be important to assess why this resistance has developed, so that an appropriate response can be developed.
Monitoring and evaluation: tools for biodiversity conservation and development projects
Why resistance matters
Resistance is any force that slows or stops movement. People resist in response to something. This could be in response to something we do, or to something we propose. Changes do not fail because of resistance—they often fail because of the way in which we respond to resistance.
Three levels of resistance can be identified.
e Level |: Based on information (‘| don’t understand it.’)
This level of resistance is based on the information, the facts, figures and ideas, the presentations and the logical arguments that are often used to persuade people that change is necessary.A typical response is to offer even more information when something completely different is called for.
e Level 2: Physiological and emotional reactions to the change (‘I don’t like it.’)
This level of resistance is based on fear—people fear that they will lose face, friends, perhaps even their jobs. This fear is physiological and can be triggered without conscious awareness. People ex- periencing these symptoms have a diminished ability to listen, and conversations can become very strained.
e Level 3: Bigger than the current change (‘I don’t like you.’)
People are not resisting the change in itself—they are resisting the source of the change. They may be resistant because of their relationship with the person or group perceived as responsible for the change, and whom they are seen to represent. Level 3 resistance is also the area of cultural, religious and racial differences. It is unlikely that people will place much value on the ideas and opinions of the other side.
The challenge in working with resistance is to become aware of the level from which the resistance is coming, and to deal with it through appropriate strategies. Level | tactics of communication will seldom work at Level 2 and 3.
(Maurer, undated; Thaw 2002)
89
90 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Working with resistance
Having read the piece on ‘Why resistance matters’, make some notes on the following:
= Where have you encountered resistance? = What was your response to this resistance?
* What do you think was the value of this resistance?
Se)
Monitoring and evaluation: tools for biodiversity conservation and development projects
w y
Different methods for decision-making in groups have been identified. The chosen method should be appropriate in the context you face, taking account of such factors as the time available, the past his- tory of the group, the kind of task on which you are working, and the kind of climate you are trying to develop in the group.
e Decision by lack of response
Group members may offer ideas for the group to consider. If there is no response to the idea, the group's lack of response is an effective decision not to support the idea.
e Decision by formal authority
The group may set up or inherit a power structure in which it is understood that someone in authority will make the decisions. While group members can generate and discuss ideas, the authority figure will make the decisions when she or he feels that they have heard enough information as a base for the decision.
e Decision by self-authorisation or minority
A small number of group members may make suggestions that result in decisions without the
consent of the majority of the members. If there are no responses to the suggestions, the silence is taken as support for the decision. This assumption may be wrong, especially if people feel reluctant to be seen as the only voice opposing the suggestion or delaying the group from making a decision.
e Decision by majority rule
Members of the group can vote on a proposal or share their views, and if there is a majority view, it is taken as a decision by the group. However, the decision may not have a high level of support if some group members feel that there was insufficient time to discuss the ideas before voting took place, or that the issue was too important to be decided on the basis of majority voting.
e Decision by consensus
Decisions by consensus occur when the group has had sufficient time to discuss the issue, with all the members feeling that they have had a fair chance to influence the decision. Communication plays an important role, as group members are more likely to support a decision with which they do not fully agree if they feel that the group has listened to their concerns.
e Decision by unanimous consent
The decision is made with the full support of all the group members. While this may be necessary in certain key decisions about the project, it will not always be appropriate. It can be a very inefficient way to make decisions, as it could delay the group from reaching a conclusion.
(Schein 1999)
92 Monitoring and evaluation: tools for biodiversity conservation and development projects
In Summary:
—
REFERENCES AND FURTHER READING
MAURER, R. Undated.Why resistance matters. http://www.beyondresistance.com/resistance_to_change.htm (accessed January 2008).
SCHEIN, E.H. 1999. Process consultation revisited: building the helping relationship. Addison-Wesley, Reading MA. THAWY, D. 1996-2002. Ideas for a Change Series. Olive Publications, Durban.
Monitoring and evaluation: tools for biodiversity conservation and development projects 93
Section 6: SHARING THE LEARNING (REPORTING)
While the notion of reporting is most commonly associated with donor requirements, most projects have a range of stakeholders who want to be kept informed and to whom the project is accountable. Keeping them up-to-date is important so that they can support critical decisions that the project might have to make in the future.
Project-level reporting serves the needs of a number of stakeholders by: = Generating the inputs required for programme-level monitoring. = Meeting the very specific requirements of small grant-funders.
* Acting as a vehicle for accountability to the community in which the project operates, both internal (staff) and external (beneficiaries, target group, interested and affected parties).
= Acting as a vehicle for communicating the learning to the project’s partners and peers.
The reporting needs of each of these stakeholders are likely to differ in respect of both format and content. Keeping them all informed could prove to be time-consuming. Our aim here is to provide you with tools to develop effective and efficient reporting systems that enable you to disseminate the information and learning assembled through the M&E process to all stakeholders, without du- plication or undue effort.
By the end of this section, you will have:
# Explored the notion of accountability and the range of stakeholders to whom project reporting is directed.
= A better understanding of what it is that the various stakeholders want to know. = A better understanding of donor reporting terminology.
= Understood how sharing the learning can contribute to improvement in your own practice.
Developed a framework for more efficient reporting within the context of your project.
94 Monitoring and evaluation: tools for biodiversity conservation and development projects
6.1 WHAT IS ACCOUNTABILITY?
Accountability is about building and maintaining relationships based on learning and mutual respect. Interacting with your target group, project partners and beneficiaries as you exchange ideas, ex- periences and offer support, provides many opportunities for practising accountability. By this we mean accounting to all stakeholders for what the project committed itself to achieve and for creat- ing opportunities for the sharing of learning during the life of the project.
Accountability is also about acknowledging the support the project has received from various stakeholders. This can take place in different ways, but the purpose is to report to the stakeholders by explaining what has been accomplished with the resources that they made available to support the project. Their support may have been in the form of people, finances, equipment and supplies, or it may have been political (lending their support and influence).
What do the stakeholders want to know?
As they are involved in the project in different ways, the interests of stakeholders will differ. Meth- ods for distinguishing between these interests during the process of developing your M&E frame- work are discussed in Section |.In this discussion we explore the specific interests of six different groups of stakeholders.
e Beneficiaries
Beneficiaries are the people who ultimately benefit from the project intervention, for example the people of the Cape Floristic Region. In some cases, particularly in the biodiversity conser- vation sector, the benefits accrue to people indirectly because project outcomes are directed at measures that safeguard the environment and resources on which people depend. In such instances, beneficiary interests can be served through the same reporting channels as for the public.
In other instances, where there are people who are direct beneficiaries of project interventions, they are likely to take a closer interest in the project. They should be kept informed on what the project has accomplished, and would be represented in the group of interested and affected par- ties, possibly along with members of the target group.
e Target group
They are the people who are targeted by the project team as a vehicle for achieving the project purpose. It is possible that a representative group or committee of the project’s target group was formed during the project establishment phase and this is the group to whom the project can report.
The purpose of communicating with the target group is to: * Report on what progress the project is making. = Inform them of any changes in the products, goods and services that the project is delivering.
= Receive their feedback on how they perceive the quality and usefulness of your support.
Information received from this group is frequently used to assess the progress and achievements of the project. They are thus a key stakeholder in the process of accountability. In evaluation processes, they are often asked for much information and given relatively little back in return.
Monitoring and evaluation: tools for biodiversity conservation and development projects 95
While it is common to send written reports to donors, reporting to the target group can best be served through face-to-face interactions. This creates opportunities for you to be creative in
the way you report to them. The most appropriate and efficient forms of reporting are likely to be:
« Visual presentations of the project’s experiences and lessons, using video or electronic pro- jection.
= Charts, diagrams and other graphic media.
In preparing the material, you should take into account the venue where the presentation will take place and whether the representatives will, in turn, be reporting back to other groups.
Annual audits are carried out with the farmers who are participating in the programme.There could be more communication with them, on an ongoing basis, about their needs, and perceptions of the benefits of the programme. There is this assumption that people will be happy if they have signed up. The management plan is based on their original list of needs. It is likely that they are still the same, but maybe we are not tapping into landowner needs. Our resources do not stretch that far.
(Conservation Partnerships Programme)
e The public
The general public can be an important audience for project accountability. Rather than re- porting simply on what has happened, this form of accountability can take place with a view to influencing the public. They can be influenced to do things that may be to the benefit of the project and its objectives. The public can be an important source of contributions, in the form of donations and volunteers. Changes in behaviour and attitudes can help shape public percep- tions about the issues that the project is addressing, thereby enhancing its broader impact.
Members of the public who are policy-makers, even if they are not connected to the project in any way, are a potentially significant audience for project reporting. They include elected officials, such as members of parliament at national and regional levels and city councillors, as well as ap- pointed officials of government departments and agencies. By bringing the project’s experiences and lessons to their attention, they can be influenced to make changes in policy, legislation and operational requirements that will benefit a much larger population than the project could reach directly.
This group is likely to be interested in: = The changes that have occurred as a result of the project's intervention.
= The impact of the project on the lives of beneficiaries and the target group.
« The potential replicability of the project.
The most appropriate and efficient forms of reporting to this group are likely to be: # Visual presentations at public meetings, using video or electronic projection. « Feature articles, reports or letters to the editor in newspapers or magazines.
= A website or page on the partner programme's website.
96
Monitoring and evaluation: tools for biodiversity conservation and development projects Collegial interest groups and peers
Reporting to collegial interest groups and peers results in sharing the lessons with others, and building support for changes in practice. Dissemination of learning might take place at different levels, depending on who was involved in identifying the lessons. Besides the sharing that oc- curs within the project team (discussed in Section 2), reporting enables learning in the following contexts.
* Across an organisation: The organisation may host a number of projects that can benefit from each other’s experience. By paying attention to sharing lessons within the organisation, there is a greater likelihood that the whole organisation can benefit from the experience of one project.
= Between organisations: Organisations that are doing similar work can benefit from one another’s experience. This can prevent organisations that work in the same sector from repeating mistakes already made elsewhere.
This group is likely to be interested in:
» The project’s findings on the reasons for any divergence between targets and achievements, and the strategy for adaptation.
» Lessons relating to methodology and management systems. » Tools that proved to be useful to the project.
» The potential replicability of the project.
Care has to be taken with the way you disseminate lessons. There is a risk that you might lose credibility in the eyes of your colleagues and partners if they think you are trying to impose lessons on them. If the lessons are to be accepted and adopted by others, it is better if they identify the relevance of lessons to their own context. This allows them to translate a general lesson to the specific situation they are facing.
Furthermore, the nature of conservation work may mean that the lessons include scientific in- formation, survey results and other forms of technical data. It is important to think about ways to integrate this information into the project findings (and project reports) so that it can be disseminated and shared with others.
Finally, the process of disseminating learning can serve as an opportunity for team learning. By bringing people together to review their experience and the emerging lessons, the whole group has an opportunity to link the learning to innovations in upcoming events and in the project plans.
We have annual interprovincial get-togethers, 3-day peer learning bosberaads.We use that to try to integrate other programme learnings.
(Conservation Partnerships Programme)
Monitoring and evaluation: tools for biodiversity conservation and development projects
e Host programme
Most projects are embedded within a host programme. Their relationship is through their com-
mon objectives: the outputs of the project contribute to the achievement of the programme’s objectives. The host programme therefore needs information on the project’s achievements to maintain its own M&E processes.
The host programme is likely to want written reports on:
* Actual achievements relative to planned targets.
97
» Any proposed changes to the logframe and an explanation of the reasons for the divergence
between targets and achievements.
The host programme may play a supportive role by facilitating the sharing of learning that occurs within its component projects, and offering assistance to projects that may be having difficulty in complying with donor reporting, in which case it would have an interest in all the reporting outputs discussed here.
Donors
It is useful to remember that donors provide financial support because your project is in line with their own objectives. They want regular written reports that describe project progress, its context and financial history. These reports help them to assess whether resources are being used in line with the plans, and what progress is taking place. They, in turn, often use this infor- mation in their own reporting to their donors and supporters.
The information that they are interested in revolves around these questions: # Is the project achieving its planned outcomes?
» Are the resources they contributed being used in an efficient and effective manner?
" Also, if the project is contemplating any major changes to the planned outputs or use of resources, what are these planned changes, and what are the reasons and motivations for making the changes?
Some ways to disseminate learning
e Communities of practice
These forums bring together people interested in learning from one another about a common area of practice. Such groups can be formal or informal, operating with or without the support of the organisation. Some communities of practice take place in person, as people share a common workplace. An increasing number of such communities are making use of the internet by creating electronic or virtual communities that permit them to interact with people doing similar work in other parts of the country and around the world.
e Websites
These are a more recent forum for disseminating project lessons. If the project or organisation has its own website on the internet, you can develop a section for ‘Emerging Lessons’. This can be an additional way of disseminating any writing you have done. You can also make contributions to websites maintained by other organisations.
98 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Newsletters, magazines and other media
Writing about your experience is another way to disseminate your learning. By writing and publishing articles in the newsletters, magazines, blogs and other media read by your colleagues, you have the opportunity to share your perspective on how your fellow professionals can improve their practice. .
e Seminars and conferences
r
1
'
'
'
'
'
'
'
'
1
1
1
1
1 You can use the opportunities provided by seminars and conferences arranged by networks and NGO coalitions as a further avenue to share your learning. You can prepare a presentation for ' such an event, or be part of a panel discussion in which you describe what you have been learning ! through your project experience. 1
!
1
1
'
1
1
1
1
1
1
1
1
1
1
1
‘
&
e Knowledge management
This is an organised way for an organisation to systematically capture and disseminate the knowledge that is created within the organisation. It can make it easier for staff to locate people and knowledge as they need it to help them solve problems they face in their work.A growing number of NGOs are now paying attention to knowledge management programmes, especially for sharing knowledge between projects and offices located in different geographic areas, and for informing
the support and policy work of a central office. Knowledge management relies on a strong culture among people in the organisation for documenting and sharing what they know with one another.
(Collison & Purcell 2004; Bellanet website)
Monitoring and evaluation: tools for biodiversity conservation and development projects 99
6.2 REPORTING FORMATS
Many donors have developed specific formats for written reports, and funding contracts often stipulate dates when reports should be submitted. It can mean much additional work for project staff if the project’s own formats for gathering information and internal reporting differ from those of the donor.
This becomes even more complicated if the project, or its host organisation, has multiple donors, each with its own reporting formats and timeframes. Many donors are sympathetic to the re- porting burden faced by project staff. ln some cases it may be possible to develop a common and agreed format for reporting that meets the accountability needs of a number of donors. For ‘small grants’ beneficiaries, reporting can be framed as qualitative commentary because donors’ specifica- tions are fairly loose.
Some examples of the reporting information required by small grants donors include: = Updates on progress made against objectives. = Evaluation of accomplishment/ successes. * Asummary describing the project’s conservation impacts. = Progress towards the achievement of the objectives. « Unexpected impacts. # Intended or unintended negative consequences. « Disappointments or failures during implementation.
It can be difficult for people who are not experienced in reporting to respond to open-ended specifications such as these, and as a result they may fail to provide the information that donors are looking for.
A framework for progress reports
L R
While many donors have developed formats for written reports, not all donors require their grantees to use a specific format. If you have flexibility in the way you report, or if you are trying to negotiate with donors for a format that is more appropriate to your needs, you might consider including the fol- lowing elements:
= Current status of the project.
= Major activities undertaken during the period under review (tasks completed or in progress within each activity).
# The project’s financial performance. = Estimates of the number of beneficiaries serviced, relative to targets, and their responses.
=" Current and potential problems (this includes a description of the problems, and remedial actions planned or recommended).
= Project plans and schedules for major activities during the next review period. This framework can assist your communication with donors and other stakeholders, and reveals the
reasons why you are considering changes. The reports can themselves be used in future monitoring, as they will serve as a summary of what you intended to accomplish during the next period.
1 1 1 1 1 1 1 1 1 1 1 1 1 1 ! i 1 1 1 1 1 1 1 1 1 1 1 j 1 1 ; (Casley & Kumar |987) 1
100 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Using the action learning cycle for report writing
Review a recent monitoring and evaluation report produced by your project.
Assess what percentage of the report fits with each of the four parts of the action learning cycle:
Experience. Reflection. Insight and understanding.
Planning. Are you satisfied with this distribution? Are any of the components absent or underrepresented?
How might you change the way you prepare and produce reports so that they include a better distribu- tion of the various elements of the action learning cycle?
Monitoring and evaluation: tools for biodiversity conservation and development projects 101
Unpacking reporting terminology
The most commonly used terms in donors’ reporting specifications are: e Progress relative to objectives
This may be phrased as ‘progress made against objectives’ or ‘progress towards the achievement of the objectives’. You will find a description of the project’s objectives in the logframe or contract with the donor. The objectives are likely to be referred to as ‘results’ or ‘outputs’. In your report, you have to list each of these as a heading and below that objectively describe all the activities that have been undertaken towards achieving that result or output.
e Evaluation of accomplishments or successes
Evaluation requires a comparison of what has been achieved against what was planned. The project plan is a useful source of reference for what was planned.Where there is a difference between the two, the evaluation should provide some insights as to why, and what can be learned from it. (Refer to Section 4.)
Below are some terms used by Small Grants Funders. Given their open-endedness, it is useful to discuss with your donors their understanding of the phrases and come to an agreement on the way you will use them in your reports.
e Conservation impacts
While ‘impact’ is sometimes used to refer to a long-term goal, in the context of project-reporting it is an enquiry into the immediate impact that the project is having in its local context. What is the project contributing immediately towards the long-term goal of biodiversity conservation?
e Unexpected impacts
Sometimes projects have positive outcomes that were not envisaged or intended in the project plan. These ‘side-effects’ or ‘by-products’ may represent important benefits and can be extremely useful learning opportunities.
e Intended or unintended negative consequences
Just as a project may generate unexpected benefits, so too can it result in unexpected negative consequences. In some cases, negative consequences may have been anticipated as intended and unavoidable outcomes of a project intervention.
e Disappointments or failures during implementation
This category of information provides the opportunity for presenting information about the ab- sence of enabling factors that may have impacted on the project’s progress or success.
The last three of the above terms represent opportunities for the reporter to explore the ‘why’ of any divergence between planned and actual outputs and lay the foundation for motivating for a change to the project plan.
102 Monitoring and evaluation: tools for biodiversity conservation and development projects
By analysing elements of the previous discussion, we can see that there is some overlap in the interests of the different stakeholders:
Public/
bene- Host pro- Reporting content ficiaries gramme Progress relative to objectives v v v Localised impact of project outputs v v Potential replicability of the project v v v Lessons relating to methodology and y yy © management systems Tools that were useful to the project v v Efficiency and effectiveness of resource ie V use
Reasons for divergence between targets and achievements, and strategies for v v v adaptation
The information needed by the host programme is a subset of the information needed by the do- nors, both of which are written reports.
Likewise, in preparing presentations for the general public, the content would be the same as for beneficiaries, with some of the more detailed information on localised and direct benefits and impacts omitted.
The sharing of information with collegial interest groups and peers is likely to be focused on learn- ing, and presented in the context of a seminar, conference or similar collegial gathering.As with any audience, it is important that the material is tailored to the needs and interests of this particular group, even though there may be overlap with material used for beneficiaries, for example. Herein lies its usefulness: this may be the only opportunity that project staff have for the kind of objectivity and reflection that is needed to derive lessons from project practice. By encouraging staff to partic- ipate in these events, and to prepare papers and articles that disseminate the learning, the organisa- tion benefits, not only from the exposure, but also from the reflection that is enabled through the process of preparing the material.
Whether reporting content and format is externally imposed or driven by communications-based criteria,a common set of normative principles apply to project reporting at all levels, inter alia:
= Reporting should draw directly from information gathered through M&E activities. Project de- velopers should not be required to report on aspects that have not been monitored. Reporting requirements thus have to become an informing aspect in the design of the M&E framework.
Monitoring and evaluation: tools for biodiversity conservation and development projects
Over-reporting leads to reporting fatigue. Reporting formats should have clearly stated objec- tives and include only fields that will actually be useful to the users of the information. The frequency and timing of reporting should coincide with related activities in the project cycle. Duplication in reporting (to the various constituencies) should be avoided.
The challenge of communicating with different audiences is to promote a dialogue between the organi- sation and its stakeholders, and to prevent reporting from being a one-way flow of information.
e Bridging communication gaps
Differences in the status and background of stakeholders, which complicate the communication process, can be addressed with time, and with appropriate attitudes and behaviour.
e Aggregating lessons
The tension between presenting broad conclusions that feed strategy and policy work, and the rich- ness that lies in the detail of local situations.
e Reaching out Drawing the attention of specific groups to new lessons in credible and relevant ways. e Communication in reverse
Feeding back to informants the decisions taken as a result of their contributions to the monitoring process.
(Barnard & Cameron 2000)
103
104 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Building a matrix of your project's reporting requirements
By analysing the requirements of each of the stakeholders to whom your project is accountable, you will be able to identify the commonalities that will enable you to streamline your reporting system.
PROJECT STAKEHOLDERS:
AREAS OF INTEREST:
Progress relative to objectives
Localised impact of project outputs
Potential replicability of the project
Lessons relating to methodology and management systems Tools that were useful to the project
Efficiency and effectiveness of resource use
Reasons for divergence between targets and achievements, and proposals for adaptation
PRESENTATION FORMAT:
WRITTEN FORMAT:
Commonalities:
Monitoring and evaluation: tools for biodiversity conservation and development projects 105
In} Summary,
In Section 6 we:
= Examined the accountability interests of the various stakeholders and identified what this means for project reporting. = Identified the common areas and hierarchy of information needed for reporting purposes. = Looked in more detail at: » Ways of disseminating learning across the organisation and between organisations. » A generic framework for reporting on progress. » The terminology commonly used in donor reporting. » Methods for communicating with your key audience.
= Provided a matrix form to help you to analyse your project’s reporting requirements.
By interacting with stakeholders and professional colleagues, you create the opportunity for re- flection and learning in your own practice and for your experience to influence the thinking and actions in other organisations. Their experience of introducing your ideas will stimulate their own reflection and learning, resulting in ongoing improvements in their practice and impact.
REFERENCES AND FURTHER READING
BARNARD, G. & CAMERON, C. 2000. Efela Synthesis Report: evaluation feedback for effective learning and accountability. Institute of Development Studies, Brighton. http://www.oecd.org/dataoecd/ | 0/29/2667326.pdf (accessed January 2008).
BELLANET Undated. Knowledge management for development organisations. http://home.bellanet.org (accessed January 2008).
CASLEY, D.J. & KUMAR, K. 1987. Project monitoring and evaluation in agriculture. Johns Hopkins University Press, Baltimore.
COLLISON, C. & PARCELL, G. 2004. Learning to fly: practical lessons from one of the world’s leading knowledge companies. Capstone, Oxford.
Monitoring and evaluation: tools for biodiversity conservation and development projects 107
This handbook has introduced a number of ideas and resources that can help you as you prepare to assess your work by engaging in a monitoring and evaluation process. We have also offered ma- terial to assist you in becoming more mindful about your practice of project M&E.
There is no magic formula that you can follow to develop an effective project monitoring system and practice. We have introduced a broad framework and outlined the elements of an approach
to your M&E practice that can be adapted to fit the needs and context of your project and your organisation.
As you explore this issue further, remember that M&E takes place to support the implementation of your project. It does so by allowing you to step back from your work occasionally to consider what you have achieved, and the changing context in which you are situated. M&E exists to serve the project—it should not develop a life of its own, independent of the information and reflection needs of the project.
Ultimately, M&E can provide you with insight and perspective. It can help you to find things in your work to value, and to identify practices to change. By reflecting on your work and your achieve- ments, you will be equipped with resources to begin a new cycle of analysis and planning, devel- oping new projects to support people in conserving biodiversity and meeting their development needs.
We hope that you have found this component of the SANBI Biodiversity Series helpful in your prac- tice and work. The ongoing cycle of planning, implementation and reflection allows us to be con-
tinual learners, both as individuals and as organisations.And this can only be to the benefit of the people with whom we work and the environments in which we work.
Monitoring and evaluation: tools for biodiversity conservation and development projects 109
FURTHER READING
There is an extensive virtual library that can provide more information, resources, approaches and tools that complement this handbook. The material presented in the handbook draws on two main traditions:
e Participatory Learning and Action (PLA)
The PLA tradition builds on the experience of participatory tools such as Participatory Rural Appraisal (PRA). It promotes the use of methods in a participatory manner, allowing respond- ents to be involved in deciding which methods are appropriate, designing them, and collecting information. By involving participants in the analysis of information for significant changes, there is a greater likelihood that project stakeholders, as well as project staff, will learn from the ex- perience.
A related practice has been developing among development practitioners in many countries in recent years. It is known as Participatory Monitoring and Evaluation (PME).
e Social Sciences
Some development practitioners and researchers are interested in people, groups and organisa- tions, and the relationships between them and their surrounding environment. They have de- veloped tools to allow them to investigate what has taken place, and what people think about these things. While the tools can collect quantitative or qualitative information, many practition- ers are more interested in qualitative information.
These two traditions are built on a strong conceptual foundation and have been improved through many years of field-based application by practitioners all over the world. However, some adaptation may be needed to ensure that they fit your specific context.
A number of toolkits and books that provide additional detail on various methods for project monitoring and evaluation have been written in recent years. They provide more extensive de- scriptions and examples than what is possible in the space available in this handbook. Readers are encouraged to make use of these resources for more explanation on the methods outlined on the previous pages.
# DflD 2002.Tools for development. London, UK.
This toolkit contains a helpful overview of approaches and techniques that can be useful for ef- fective project implementation. Available at www.dfid.gov.uk/pubs/files/toolsfordevelopment.pdf.
= FEUERSTEIN, M.T. 1986. Partners in evaluation: evaluating development and community pro- grammes with participants. Macmillan, London.
This book provides a wide range of methods for participatory monitoring and evaluation. Writ- ten in an easy-to-understand format, with clear illustrations.A subsidised edition of the book is available at low cost from www.talcuk.org.
= GUIT, |. 1998. Participatory monitoring and impact assessment of sustainable agriculture ini- tiatives. SARL Discussion Paper No. |. International Institute for Environment and Development (IIED), London.
This handbook describes the author’s experience in developing key elements of an approach to project monitoring. It details 20 methods, and is available from www.iied.org.
110
Monitoring and evaluation: tools for biodiversity conservation and development projects
IUCN. PME course documents.
Available at http://www.iucn.org/themes/eval/documents2/pme_concepts/pme_short_train- ing course_OlI.pdf.
IUCN. Evaluation handbook (on managing an external evaluation).
Available at http://www.iucn.org/themes/eval/documents2/eval_handbook/handbook_eng.pdf.
Material made available by the Network of Networks on Impact Evaluation (NONIE).
http://www.worldbank.org/ieg/nonie/docs/IE_statement_draft.doc. http://www.worldbank.org/ieg/nonie/docs/guidelines subgroup 2.doc.
ODI 2005. Successful communication. London, UK.
The toolkit lists a variety of practical approaches that can be used for communicating a project’s purpose and messages to key stakeholders and beneficiaries. Available at http://www. odi.org.uk/RAPID/Publications/Documents/Comms_toolkit.pdf.
ODI 2006. Tools for Knowledge and Learning. London, UK
The booklet offers a range of tools that can be used by a project team throughout the project implementation process to help with internal management and quality improvements. Available at http://www.odi.org.uk/RAPID/Publications/Documents/KM_toolkit_web.pdf.
PRETTY, J.N., GUIT, l., THOMPSON, J. & SCOONES, |. 1995. Participatory learning and action: a trainer’s guide. International Institute for Environment and Development (IIED), London.
An introduction to PLA, the guide also includes 100 exercises that can be used in working with groups, gathering and analysing information, and for decision-making. Available from www.iied. org.
WADSWORTH, Y. 1997. Do it yourself social research, edn 2.Allen & Unwin, St Leonards.
This is the second edition of a popular Australian manual, full of practical advice on how to use many of the common social science research tools, such as interviews and surveys. Available from www.allen-unwin.co.au.
Monitoring and evaluation: tools for biodiversity conservation and development projects It
BIBLIOGRAPHY
ABBOTT, J. & GUIJT, |. 1998. Changing views on change: participatory approaches to monitoring the environment. SARL Discussion Paper No. 2. International Institute for Environment and Development, London. http://www.iied.org/ pubs/pdfs/6 | 40IIED.pdf (accessed January 2008).
ACTION RESEARCH. http://www.scu.edu.au/schools/gcm/ar/arp/arphome.html (accessed January 2008).
BARNARD, G. & CAMERON, C. 2000. Efela synthesis report: evaluation feedback for effective learning and accountability. Institute of Development Studies, Brighton. http://www.oecd.org/dataoecd/ | 0/29/2667326.pdf (accessed January 2008).
BELLANET. Knowledge management for development organisations. http://home.bellanet.org (accessed January 2008).
CASLEY, DJ. & KUMAR, K. 1987. Project monitoring and evaluation in agriculture. Johns Hopkins University Press, Baltimore.
CAVILL, S. & SOHAIL, M. 2007. Increasing strategic accountability: a framework for international NGOs. Development in Practice 17,2.
CDRA 2001. Measuring development: holding infinity: a writing from the CDRA Annual Report 2000/2001. http://www. cdra.org.za (accessed January 2008).
CIDA 1997. Guide to gender-sensitive indicators. http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Policy/$file/ WID-GUID-E.pdf (accessed January 2008).
COLLINGWOOD, C. 2007. Revealing practice, re-imagining purpose, claiming our place Biennial Practice Conference, McGregor, May 2007. http://www.biennial.cdra.org.za (accessed January 2008).
COLLISON, C. & PARCELL, G. 2004. Learning to fly: practical lessons from one of the world’s leading knowledge companies. Capstone, Oxford.
DAVIES, R. 1997. Lesson learning: how will we recognise it when we bump into it? http://www.mande.co.uk/docs/ lessons.htm (accessed January 2008).
DE RUIJTER,A., DIETZ, T..VAN GONGEN, E., HELMSING, B. & KNORRINGA, P. 2006. Evaluation of the Theme-based Co-financing Programme. Cross-cutting study: monitoring and evaluation. Prepared for Ministry of Foreign Affairs of the Netherlands. http://www.tmf-evaluatie.nl/eindrapportagem20M&E.doc (accessed January 2008).
ELDIS. Participatory monitoring and evaluation. http://www.eldis.org (accessed January 2008).
ESTRELLA, M. (ed.) 2000. Learning from change: issues and experiences in participatory monitoring and evaluation. International Development Research Centre, Ottawa.
ESTRELLA, M. & GAVENTA, J. 1998.Who counts reality? Participatory monitoring and evaluation: a literature review. IDS Working Paper 70. Institute of Development Studies, Brighton. http://www.ids.ac.uk/ids/bookshop/wp/wp/70.pdf (accessed January 2008).
FEUERSTEIN, M.-T. 1986. Partners in evaluation: evaluating development and community programmes with participants. Macmillan, London.
FOWLER, A. 2000. The virtuous spiral: a guide to sustainability for NGOs in international development. Earthscan, London.
GOOLD, L. 2006. Working with barriers to organisational learning. Bond, London. http://www.bond.org.uk/pubs/briefs/ olbarriers.pdf (accessed January 2008).
GOSLING, L. & EDWARDS, M. 1995.Toolkits: a practical guide to assessment, monitoring, review and evaluation. Development Manual 5. Save the Children, London.
GUIJJT, |. 1998. Participatory monitoring and impact assessment of sustainable agriculture initiatives: an introduction to key elements. SARL Discussion Paper No. |. International Institute for Environment and Development, London. http://www.iied.org/pubs/pdfs/6 | 39IIED.pdf (accessed January 2008).
HARRISON, R. 1995. The collected papers of Roger Harrison. McGraw-Hill, New York.
IDS 1998. Participatory monitoring and evaluation: learning from change. IDS Policy Briefing | 2. http://www.ids.ac.uk/ UserFiles/File/publications/policy_briefs/PB|2.pdf (accessed January 2008).
INTERNATIONAL INSTITUTE FOR ENVIRONMENT AND DEVELOPMENT. Participatory resources. http://www.iied. org (accessed January 2008).
KELLEHER, D., McLAREN, K. & BISSON, R. 1996. Grabbing the tiger by the tail: NGOs learning for organizational change. Canadian Council for International Co-operation, Ottawa.
112 Monitoring and evaluation: tools for biodiversity conservation and development projects
KNOWLES, M. 1980. Cited on the Quay Communications website. www.quaycom.com/library_images/white_papers/ adult_learning.doc (accessed January 2008).
LAWRENCE,A. (ed.) 2002. Participatory assessment, monitoring and evaluation of biodiversity. ln Summary of the ETFRN internet discussion 7—25 January 2002. http://www.etfrn.org/etfrn/workshop/biodiversity/documents/ dissum.doc (accessed January 2008).
LIEB, S. 1991. http://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/ teachtip/adults-2.htm (accessed January 2008).
MONITORING AND EVALUATION. M and E News. http://www.mande.co.uk (accessed January 2008).
MAURER, R. No date.Why resistance matters. http://www.beyondresistance.com/resistance_to_change.htm (accessed January 2008).
PRETTY, J., GUIJT, |, THOMPSON, J. & SCOONES, |. 1995. Participatory learning and action: a trainer's guide. International Institute for Environment and Development, London.
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ. RANDEL, M. 2002. Planning for monitoring and evaluation. Olive Publications, Durban.
REELER, D. 2001. Unlearning—facing up to the real challenge of learning. http://www.cdra.org.za (accessed January 2008).
ROGERS, PJ. & WILLIAMS, B. 2006. Evaluation for practice improvement and organisational learning. In |. Shaw, J. Greene & M. Mark (eds), Handbook of evaluation. Sage Publications, London.
SCHEIN, E.H. 1999. Process consultation revisited: building the helping relationship. Addison-Wesley, Reading MA. SENGE, P. 1994. The fifth discipline fieldbook. Nicholas Brealey, London.
SENGE, P. 1999. The dance of change: the challenges to sustaining momentum in learning organisations. Currency/Doubleday, New York.
SHAPIRO, J. 1995. Financial management for self-reliance: a manual on managing the finances of a non-profit organisation. Olive Publications, Durban.
SHAPIRO, J. 1996. Evaluation: judgment day or management tool? Olive Publications, Durban. SMILLIE, |. & HAILEY, J. 2001. Managing for change: leadership, strategy and management in Asian NGOs. Earthscan, London.
SOAL, S. 2001. How do we know what difference we are making? Reflections on measuring development in South Africa.Adapted from a presentation to a workshop between Christian Aid and South African partners, Durban, October 2001. http://www.cdra.org.za (accessed January 2008).
SOAL, S., HILL, S. & REELER, D. 2007. Evaluation: a developmental approach. http://www.cdra.org.za (accessed January 2008).
STAKE, R.E. 1995. The art of case study research. Sage Publications, California.
STEM, C., MARGOLUIS, R., SALAFSKY, N. & BROWN, M. 2005. Monitoring and evaluation in conservation: a review of trends and approaches in conservation biology. Blackwell, Gainesville Florida.
TAYLOR, J. 1998. NGOs as learning organisations. http://www.cdra.org.za (accessed January 2008).
TAYLOR, J., MARAIS, D. & KAPLAN, A. 1997. Action learning for development: use your experience to improve your effectiveness. Juta, Cape Town.
THAW, D. 1996-2002. Ideas for a Change Series. Olive Publications, Durban.
UPHOFF, N. & COMBS, J. 2001. Some things can’t be true but are: rice, rickets and what else? http://www.eldis.org/ fulltext/TRUE-RRR.pdf (accessed January 2008).
WADSWORTH, Y. 1997. Do it yourself social research, edn 2.Allen & Unwin, St Leonards. WADSWORTH, Y. 1997. Everyday evaluation on the run, edn 2.Allen & Unwin, St Leonards.
WOODHILL, J. 2006. Monitoring and evaluation as learning: rethinking the dominant paradigm. In J. de Graaff, J. Cameron, S. Sombatpanit, C. Pieri & J. Woodhill (eds), Monitoring and evaluation of soil conservation and watershed development projects. World Association of Soil and Water Conservation. Science Publishers, New Hampshire, USA.
WWE (World Wide Fund for Nature) 2005. Basic guidance for step 2.2: monitoring plan. http://assets.panda.org/ downloads/2_2_monitoring_plan_01_11_O5.pdf (accessed January 2008).
YIN, R.K. 2003. Case study research: design and methods. Sage Publications, California.
Monitoring and evaluation: tools for biodiversity conservation and development projects 113
Appendix 1: GUIDELINES FOR TRAINERS
INTRODUCTION
In this guide we suggest ways of using it for the purpose of training and building the capacity of others in M&E. In particular, it is aimed at project managers who are interested in building the M&E capacity of project staff. It is also intended for project developers who are helping to support new project managers in developing their M&E skills and understanding.
The guidelines are presented as exercises and process tips for dealing with key topics in each sec- tion of the handbook.
The first, self-reflective introductory exercise is intended to immerse project managers and staff in their own experience of M&E, before they start to develop a plan for the project. It is also an exercise that project developers could use for themselves.
Exercise
Think back over your personal and working life, trying to identify all experiences of monitoring and evaluating. Consider the following questions:
When have you been an active participant in M&E? When have you been a recipient? What were these significant events, e.g. exams, driving test, performance appraisal? What were the roles you played?
Classify the evaluation into types, e.g. internal, external or self-evaluation.
What were the contexts of each evaluation?
Map this information out in whatever way works best for you, e.g.a timeline or a mind map.
Make notes on these questions:
= = What have | learnt about M&E during my life?
= What claims can | make about M&E (things | know to be true)?
= What questions do | have?
(Source: Project Developers’ Forum, |8—19 March 2008)
114 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 1: DEVELOPING AN Mae PLAN
There are a number of different themes that can be explored in this section. We have identified four key themes.
e Theme |—Working with questions as a basis for developing a plan
Here we focus on developing an M&E plan, using questions to guide the formulation of the plan. The questions encourage thinking and in-depth inquiry about what the project is generally try- ing to achieve, as well as the specific components of M&E. For example, who will co-ordinate M&E? How? When? and so on.
It is likely that even if someone is relatively new to M&E, they will have some experience to draw on. Furthermore, for project managers who are formally adopting an M&E system for the first time, there are likely to be existing practices in the project that can feed directly into this framework.
Therefore a first step in supporting someone who is developing an M&E plan is to help them to identify what is already in place, and what is currently happening.
Exercise
Introduce the eight key questions identified in the handbook Section |: Developing an M&E plan, page 12.
Provide a brief explanation of what each question is asking for. In some cases, you might have to give reminders about terminology. For example, when looking at the objectives of the project, you could use this as an opportunity to clarify the ‘hierarchy’ of objectives that are used in the Logical Framework Approach (LFA). For question 2 (Who will use the information collected?), you could help them understand the different categories of end-users by naming them, i.e. beneficiaries, target group, project staff, management and donors. For question 3 (How are indicators used and developed?), you might check that participants understand the difference between quantitative and qualitative indicators, and so on.
Ask the individual or group to spend one hour working on their own, or in pairs, making brief notes on what is currently in place in their project regarding the eight questions.
Debrief the exercise by asking:
* How well do we understand our project? Are there any areas where we would benefit from
knowing more about our project? (‘Reviewing your objectives’ on page 16, could also be used to check understanding of the project.)
What are the strengths of our existing M&E activities? Where are there gaps?
What can we do to address these gaps?
Who can help us with this?
By when?
e Theme 2—Developing indicators
Developing indicators requires rigour and creativity. Rigour ensures that indicators are precise and measurable, and creativity is needed to identify indicators that can be measured in the required time, with the available resources and capacity.
Monitoring and evaluation: tools for biodiversity conservation and development projects 115
Before you begin, check that there is shared understanding of LFA terms within the project. (It is likely that project staff will have some familiarity with the language.) Draw out their under- standing of the key terms: objectives, outcomes, outputs and indicators.
From this basis, it is possible to develop useful project indicators.
Exercise
|. Explore the different aspects of quantitative and qualitative indicators, generating examples of each.
2. Identify the level of project objectives for which you want to develop indicators.
3. Follow the guidelines on ‘How do you develop indicators?’ on page 24.
e Theme 3—Unpacking the notion of ‘participation’
This theme helps project managers and staff to understand what participation means for differ- ent stakeholders in the M&E process, and how this can be better managed. Appropriate plan- ning of participation is important so as not to overburden stakeholders, or alternatively, to miss out on vital information and opportunities for analysing and acting on it in a participatory way.
Exercise
Ask for examples of the benefits of adopting a participatory approach to M&E.Ask for examples where participation of stakeholders can be negative.
Display a familiar example of a project matrix. Present the different phases of the M&E process: Develop the plan. Gather the information. Analyse the information. Act on the analysis. Share the learning. Develop the matrix on page 32 of the handbook interactively, on a white board or flipchart paper. Ask these questions during this process: Who should participate at the different points in the M&E process? What can they contribute? How will it benefit them? Will they be end-users of the information gathered? Will specialised technical capabilities be needed? What can be done to ensure maximum participation of different stakeholders? When will the different phases in the M&E process happen? Is specific information needed for regular management meetings? Is the sequencing of different activities important? Are there set review dates? When are informants available?
Are project staff available to carry out M&E work?
Discuss participation in the M&E process using this stimulus question: ‘What are the management responsibilities for ensuring participatory M&E?’
116 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Theme 4—The action learning cycle
Learning from experience is the desired outcome of our ongoing work, and lies at the heart
of a good M&E system. This learning does not happen automatically. Rather it requires that we take some time to look back, to ask important questions, to make meaning of the data gathered through reflection, to plan to make changes in our practice and then to implement these chang- es. The action learning cycle is one model of learning that provides a set of steps to ensure that learning does take place as a result of our experience.
Exercise
It is often said that ‘we learn from our experience’. Discuss this with the group and then request the individual or group to identify ‘a slice of experience’, or ‘an incident in time’, that is common to all in the group. It must arise from their work situation.
Ask the individual or group to write down a description of what happened, and then share it. Collaboratively build a description of the incident on a flipchart.
Pose reflection-type questions, referring to ‘the action learning cycle’ on page 37 of the handbook. Again, note the answers on the flipchart.
Facilitate a conversation about what insight and understanding can be drawn from what happened.
Ask individuals to draw up a plan to act on the insights gained.
Ask individuals to describe the planned actions to one another.
Present the diagram of the action learning cycle. Relate it to the steps just followed and to the M&E process.
Monitoring and evaluation: tools for biodiversity conservation and development projects 117
Section 2: LEARNING
This section of the handbook further develops the conceptual and practical aspects of learning within an M&E context.
e Theme |I—Learning concepts
Understanding how learning happens and exploring some learning concepts helps us to be more conscious of the way we think and process information received via our different senses. These patterns of processing could either enhance or limit learning, and so impact on our moti- vation to change and develop our practice in the field.
A first step in encouraging our openness to ‘learning through doing’ is to understand our men- tal models.
Exercise
&
y
‘SS
Display a simple picture of the outline of two people facing one another. Inside each of their heads is a completely different shape—one has a whirl shape, the other a daisy shape with petals attached to a clear centre.A common thought bubble reads, “You think funny!’ Have a conversation about how different thought patterns could help or hinder learning and change.
Present a brief outline of Single-loop learning and generate some examples from joint experience. Present a brief outline of Double-loop learning and generate some examples from joint experience.
Display two complex diagrams. One must be an example of known subject material such as an endangered plant species, and the other unknown material such as a combustion engine.Ask the individual or group to jot down their thought responses to each of the two diagrams. Write up the two sets of thoughts on separate flipcharts.
Make a brief presentation on Patterning and Puzzling, relating each to responses in the previous step.
Discuss how to go about facilitating learning in the project team. Finally, allow some time for individuals to consolidate their understanding by drawing up a set of personal facilitation guidelines.
.) , Ra eo .
iy \
118 Monitoring and evaluation: tools for biodiversity conservation and development projects e Theme 2—Tools for learning in the project team
The project manager also has to be the leader of learning in the project team. Being familiar with, and having the confidence to use, a variety of learning tools makes it easier for project managers to facilitate team learning and development.
Exercise
Request project manager/s to relax, close their eyes and imagine that they are about to learn about some amazing new phenomenon in the environment.Ask some leading questions to help the imaginative journey. Where are you? What is happening around you? What do you see? What are you thinking? What is this amazing new thing?
Ask them to open their eyes.
Provide a sheet of paper and a set of oil pastels.Ask them to draw the phenomenon. Ask them to work quickly and draw what comes to mind immediately rather than a detailed, precise picture. Invite the sharing of the pictures, and the exchange of comments.
Then ask, What inspired and informed the ‘thing’ or phenomenon you imagined? What helped you to imagine this? What in your background (your discipline, experience, skills) led you to imagining this phenomenon?
Open up a discussion about how, consciously or unconsciously, we draw on a particular discipline or field of expertise to notice our environment, imagine new things and understand the world.
Ask manager/s to reflect on the steps above and then list the different concepts, tools and frameworks they used—or would like to have used—to enhance the learning experience. Display the list/s and note common and unique tools listed. Use coloured markers to highlight the differences.
Make a brief presentation based on the information found on page 43, ‘Using case studies to learn about your conservation practice’.
Brainstorm a list of different learning tools.Add these to the list if they were not mentioned:
Peer assistance. Debriefing. Team meetings and workshops. Bench marking. Invite a brief description of these tools and refer participants to the notes on pages 45 and 46.
Allow time for managers to talk together to clarify opportunities to use the different team learning tools in the future.
Conduct an ‘after action review’ of this capacity-building session and conclude by referring to the notes on page 46.
e Theme 3—How to encourage learning
It is important to identify and understand the different factors that encourage learning and to ensure that they are in place in the project team. It is just as important to know the blockages to learning, and to have the capacity to clear or reduce the impact of these hindering factors.
The busy nature of conservation work can make it difficult to find the time to stop and learn. So while we often go through the technical steps required to monitor and evaluate project progress, individual and team learning can get neglected.
Monitoring and evaluation: tools for biodiversity conservation and development projects 119
Exercise
Make conscious what drives our learning.
Request project manager/s to reflect individually and identify for themselves the internal and external drivers of their learning. Out of this personal reflection, ask them to further consider what they think drives their team members’ learning.
Factors that encourage learning.
Make a brief presentation, drawing from the notes on page 50, ‘Building a learning environment’. Ask project manager/s to assess which of these factors that encourage learning exist in the project/s they manage.
How to ensure that the six enabling factors are present in the project team.
Ask what steps have to be taken to strengthen these factors.Allow time for individual work. Share the steps and refine a set for each project manager.
Overcoming the barriers to learning.
Hold a conversation about ways to overcome the two barriers to learning (strong negative emotions and a bias-for-action).
Action steps to overcome barriers to learning.
Invite individual/s to work on action steps that they, as project managers, could take to overcome barriers to learning.
120 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 3: GATHERING THE INFORMATION
e Theme |—Different types of information
Information is the fuel that drives project M&E systems. It is important to gather the informa- tion that is needed to feed into timely and relevant decision-making, reporting and learning. The project plans should specify what should be monitored through the information gathering proc- ess and the sources, or means of verification (MOYs), of this information.
Different types of information are needed to measure objectives at the level of outcomes and outputs, as well as to track what is happening to the assumptions.
Exercise
Information relating to outcomes.
S
Present the characteristics of this information and provide some examples of different impact reports. Hold a conversation about what is required to gather this kind of information.
Information relating to outputs.
Present the characteristics of this information, then discuss some of the problems encountered with internal documentation, and possible solutions to this. Take into account issues of accuracy, lack of uniformity, and the challenge of keeping a balance between quantitative and qualitative information.
Information relating to assumptions.
Explore the importance of information that takes into account the presence or absence of external factors on which the success of the project is dependent. Invite project managers and staff to give examples of projects suffering the consequences of being unaware of changes in external factors. Brainstorm ways to gather the relevant information and possibly influence the environment.
Qualitative and quantitative information.
Call for examples of qualitative information and list these on a flipchart. Repeat the exercise for quantitative information.Ask participants to provide their definitions of both types of information. Hold a conversation about the pros and cons of these different types of information.
Primary and secondary information.
Present the characteristics of both. See if there are any examples from their own experience that participants can identify. Tabulate advantages and disadvantages of both and then explore ways to overcome the challenges.
e Theme 2—Techniques for gathering information
There are a wide variety of techniques to gather information and it is important to expand the project manager/s repertoire.A clear set of criteria to help select the most appropriate tech- nique for a specific monitoring task will help them to make decisions about which techniques to use in which circumstances. Each discipline has its own specific set of tools and techniques. Draw attention to this, highlighting that the tools in this section of the handbook, and in the handbook in general, come from the social sciences, and some specifically from the tradition of participatory learning and action (PLA).
Monitoring and evaluation: tools for biodiversity conservation and development projects 121
Exercise
Favoured and familiar information gathering techniques.
Request project manager/s to identify the three techniques they favour and use most often. Ask them to develop a brief set of pros and cons for each. Briefly discuss these ideas.
Expanding the repertoire.
Present a list of 15 different techniques, giving a brief description of each. (Take note of which techniques have already been explored in the previous step and adjust the presentation.) Selecting the appropriate technique for the task.
Present or generate a number of monitoring task scenarios and request the project manager/s to select appropriate techniques from the list, using the selection criteria on page 57 of the handbook (Timeline, Relevance and Cost-effectiveness).
Increasing understanding of different techniques.
Request project manager/s to take time to read through ‘A toolbox of techniques for gathering information’ on pages 59 to 62. Divide the list into appropriate sections and allocate to small groups. If working with an individual, the different sections would be worked with according to a schedule. For each section:
» Discuss the methods » Develop a set of pros and cons for each
» Use the monitoring scenarios from Step 3 as a basis to design an example for each of the meth- ods
» Write up the design on a chart
» Share designs and invite feedback and suggested improvements.
Summarising.
Request project manager/s to read the summary of ‘Techniques for gathering information’ on page 58 and add to ‘Other factors to consider ’, based on their experience, and new insights gained.
122 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Theme 3—Information management systems
In some projects, the act of gathering monitoring information becomes part of the habitual pattern of the work process, and its purpose and further use is not considered, or gets forgotten. Dusty piles of evaluation questionnaires administered after a specific intervention are a common sight in many project offices. It is important for a coherent information management system to be designed and put in place.
Exercise
Improving systems already in place. Ask project manager/s to reflect on the project they manage:
* Identify all the monitoring processes that gather information at all the levels: Outcomes level, Outputs level, Activities level, Input/Resource usage level
In what form is this information collected?
Who uses it and for what purpose?
How do all the pieces link up?
How is it stored and accessed when needed?
What improvements are needed to the system? What does managing the information system involve?
Present a set of issues that need to be managed (draw from the notes on pages 63 and 64 of the handbook) and then request managers to suggest what management interventions are required.
Exploring issues of quality of information.
How does one ensure the quality of information gathered? A series of small groups can work with these issues below, and views collected in a plenary session. (Draw on notes on pages 64 and 65 of the handbook).
Accuracy. Representative of the whole picture (sampling). Overcoming potential bias. Comparability. Consistency. Focus on storage and retrieval systems. Why is a storage and retrieval system important? Present key reasons: = Comparison purposes. = Impact studies.
= Future needs analyses.
Discuss the pros and cons of different storage and retrieval systems.An expert in the field of information storage and retrieval could be invited to be a resource in this session.
Design of an information management system.
Using an example of a project plan, request manager/s to design an information management system for the project. Share designs and comment.
Monitoring and evaluation: tools for biodiversity conservation and development projects 123
Section 4: ANALYSING THE INFORMATION
e Theme I—Using questions for analysis
Questions provide a critical starting point for analysing information. Being able to formulate good, insightful questions is a key skill for project managers.
Exercise
Ask project managers and/or staff to imagine that they are someone coming from outside to evaluate the project.
Divide them up into three groups: outcome objectives, output objectives and assumptions/enabling factors.
Invite each group to brainstorm the types of questions that could be asked to establish progress towards achieving these objectives, and to understand what the contributing factors are. Similarly, what questions need to be asked to assess assumptions?
e Theme 2—Classifying evaluations
There are many different ways to classify and understand evaluations. One simple distinction is that of external, internal and self-evaluation. Beyond this, there are further categories and clas- sifications. It is a good idea for project managers to be familiar with the range of formal evalua- tions available, and the terms and concepts that are used.
Exercise
4 AY
|. Share the different types of evaluations listed on page 76 of the handbook, and check understanding.
2. Ask whether their experience of these evaluations has been as external, internal or self-evaluation. Make a list of which experiences fall under which category. Note if their experiences predominantly fall into one category.
Based on these three categories, ask participants to brainstorm the different circumstances that are best suited to external, internal and self-evaluation. Because evaluations are predominantly external, push them to consider those situations in which it would be valuable to conduct internal and self- evaluations.
e Theme 3—Contracting external evaluators
The process of contracting consultants and in this case, external evaluators, is something that project managers in the conservation sector are frequently faced with. It can be a complex, and sometimes fraught, process if it is not carefully thought through.
There are a number of aspects of the contracting process that need to be managed. These include:
« Being clear on the objectives of the evaluation. « Identifying a suitable consultant or team of consultants.
= Getting buy-in to the evaluation from internal and external stakeholders.
124 Monitoring and evaluation: tools for biodiversity conservation and development projects
* Agreeing on the approach and methodology for the evaluation. * Managing the budget. «= Managing the time frames.
* Managing the consultants.
Exercise
Debrief project managers and project staff on their previous experience of evaluations. This helps them to clarify what works and what does not work for them, and how they might like to do things differently in the future.
The ‘Sample terms of reference for an external evaluators’ on page 78 of the handbook provides a very thorough example of the different elements to consider.
2. Part of the contracting process is developing this TOR collaboratively with project staff and other important stakeholders. Assist the project manager in designing an inclusive process for the development of a TOR.
Monitoring and evaluation: tools for biodiversity conservation and development projects 125
Section 5: ACTING ON THE ANALYSIS
e Theme |I—Identifying recommendations and change
If an external evaluation has been conducted, then traditionally the recommendations for the project are generated by the external evaluators, with varying degrees of input from project staff. Increasingly, evaluations are being held and managed internally by the project itself. This provides an opportunity for project staff to really own and understand the issues that they are being faced with (internally and externally) and to generate their own solutions to these issues. Ensuring this kind of participatory approach to the evaluation process requires building it into the TOR (see Section 4 of the handbook). It also requires that project teams have the capacity to generate ideas, recommendations and solutions collectively.
Exercise
Drawing on the simple framework on page 84, ‘Keep doing? Stop doing? Start doing?’, encourage the project manager to facilitate a discussion with project staff to generate some ideas and solutions to a particular problem the project is facing.
Debrief the project manager after the team session, identifying what they did well and where they require further support.
e Theme 2—Change and resistance
It helps for project managers to know that resistance is a normal and healthy part of the proc- ess of introducing and making change.As project managers become better able to identify resistance and work with it, their efforts at making change in their project will become more effective.
Exercise
Using the resource on page 89 of the handbook, ‘Why resistance matters’, ask project managers to generate their own understanding of the reasons for, and source of, resistance.
Work with them to identify and design a strategy for dealing with a significant form of resistance that they are encountering in their project.
126 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 6: SHARING THE LEARNING
e Theme I—An expanded view of accountability
Accountability is often seen only in terms of reporting to donors. In this handbook, we have tried to broaden people’s notion of accountability. For example on page 94 in the handbook, we include not only upward accountability in our definition, but also downward, horizontal and inward accountability.
Exercise
Share the definition of accountability (page 94 of the handbook) with project staff and discuss the implications of this kind of multiple accountability.
S
Draw out what excites people about this, as well as what some of their fears and concerns might be.
Another practice that we have encouraged in this handbook in relation to accountability is to link it to learning—that we are accountable when we share the learning that the project has generated through its experiences. However, accountability and learning can sometimes be at odds with each other, particularly when accountability is equated with reporting. It becomes more difficult to share the mistakes (what did not work) and then the facts go underground.
Exercise
Ask project staff how they would like to make the link between learning and accountability. What do they think will work in their context or their environment? Where can they take some risks and be more radical in their approach?
S
e Theme 2—Making reporting accessible
Instead of seeing reporting as a chore, it can be seen as a valuable opportunity to create space for reflection—a chance to really come to grips with the challenges and opportunities of the project. However, to have the space and confidence to view reporting in this way means that project managers and project staff need to understand the basics of reporting.
Exercise
In Section 6 of the handbook, we provide a basic framework for progress reports (page 99) and we unpack some common donor terminology (page 101). Use this material to open up a discussion with participants about their current reporting requirements. Establish where they are confident, and where they require further support.
Agree on steps that can be taken to build their skills and capacity, and identify who can assist in this process, by when.
Finally, the exercise on page 104, ‘Building a matrix of your project’s reporting requirements’, can also be used to streamline the project’s reporting activities.
LA) ab = i “
Demco, Inc. 38-293
MCZ ERNST MAYR LIBRARY
wi iin
Biodiversity Series
. SANBI Biodiversity Series
a —
. Conserving South Africa’s plants: a South African response to the Global Strategy for Plant Conservation. 2006. C.K. Willis (ed.). ISBN 1-919976-28-0.
2. The Convention on Biological Diversity: biodiversity, access and benefit-sharing. A i resource for teachers (Grades 10-12). 2006. A. Solomon & P. le Grange. 7 ISBN 1-919976-30-2.
3. The Convention on Biological Diversity: biodiversity, access and benefit-sharing. A resource for learners (Grades 10-12). 2006. A. Solomon & PR. le Grange. ISBN 1-919976-31-0.
4. Fynbos Fynmense: people making biodiversity work. 2006. A. Ashwell, T. Sandwith, M. Barnett, A. Parker & F Wisani. ISBN 1-919976-29-9.
5. A plan for phylogenetic studies of southern African reptiles: proceedings of a workshop held at Kirstenbosch, February 2006. 2006. W.R. Branch, K.A. Tolley, M. Cunningham, A.M. Bauer, G. Alexander, J.A. Harrison, A.A. Turner & M.F. Bates.
ISBN 1-919976-33-7.
6. Invasive alien flora and fauna in South Africa: expertise and bibliography. 2007. C.F Musil & |.A.W. Macdonald. ISBN 978-1-919976-35-8.
™N
. Project planning: tools for biodiversity conservation and development projects. 2007. Cape Action for People and the Environment. ISBN 978-1-919976-37-2.
8. Bird Checklist for South Africa’s National Botanical Gardens. 2008. C.K. Willis, O.E. Curtis & M.D. Anderson (compilers). ISBN 978-1-919976-41-9.
9. Guidelines for Offshore Marine Protected Areas in South Africa. 2008. K. Sink & C. Attwood. ISBN 978-1-919976-43-3.
10. User profiles for the South African offshore environment. 2008. L. Atkinson & K. Sink. ISBN 978-1-919976-46-4.
11. Monitoring and evaluation: tools for biodiversity conservation and development projects. 2008. Cape Action for People and the Environment. ISBN 978-1-919976-47-1.
ont. Anic al publications by the South African National Biodiversity Institute
journal of botanical research. eg
a series of occasional publications on the southern African flora and
ray = QO Flowering Plants of Africa, a serial presenting colour plates of African plants __ with accompanying text.
OQ Flora of southern Africa (FSA), taxonomic treatises on the flora of South : Africa, Lesotho, Swaziland, Namibia and Botswana. Contributions also appear in Bothalia.
Enquiries
SANBI Bookshop, Private Bag X101, Pretoria, 0001 South Africa. Tel.: +27 12 843-5000 Fax: +27 12 804-3211 e-mail: bookshop@sanbi.org Website: www.sanbi.org.
4q ‘
ISBN 978-1-91997b-47-1
AM |
976471")
G L ) B A L national biodiversity institute
SAN BI
ENVIRONMENT
FACILITY PARTNERSHIP FUND