N
—
\ & g =
| 5 + / C.
Va a i
| |
|
j & Ba
‘ > |
== 2 :
g y | |
. 7
?
'
tools for /
aoersity conservation
and
t projects
r | Yorary
ave ZOOM)
ar para i nye vty
Muse
SANBI Biodiversity Series 11
Monitoring and evaluation:
tools for biodiversity
conservation and development
projects
The second in a series of project management handbooks, dealing with
monitoring and evaluation as a pathway to learning in a people-centred
development context
compiled by
Cape Action for People and the Environment
Pretoria
2008
SANBI Biodiversity Series
The South African National Biodiversity Institute (SANBI) was established on 1 Sep-
tember 2004 through the signing into force of the National Environmental Manage-
ment: Biodiversity Act (NEMBA) No. 10 of 2004 by President Thabo Mbeki. The Act
expands the mandate of the former National Botanical Institute to include respon-
sibilities relating to the full diversity of South Africa’s fauna and flora, and builds on
the internationally respected programmes in conservation, research, education and
visitor services developed by the National Botanical Institute and its predecessors
over the past century.
The vision of SANBI: Biodiversity richness for all South Africans.
SANBI’s mission is to champion the exploration, conservation, sustainable use,
appreciation and enjoyment of South Africa’s exceptionally rich biodiversity for all
people.
SANBI Biodiversity Series publishes occasional reports on projects, technologies,
workshops, symposia and other activities initiated by or executed in partnership with
SANBI.
Technical editor: Emsie du Plessis
Design & layout: Daleen Maree
Cover design: SANBI Graphics
————
| How to cite this publication
CAPE ACTION FOR PEOPLE AND THE ENVIRONMENT 2008. Monitoring and
evaluation: tools for biodiversity conservation and development projects. SANBI
Biodiversity Series 11. South African National Biodiversity Institute, Pretoria.
ISBN 978-1-919976-47-1
© Published by: South African National Biodiversity Institute
Obtainable from: SANBI Bookshop, Private Bag X101, Pretoria, 0001 South Africa.
Tel.: +27 12 843-5000. E-mail: bookshop@sanbi.org. Website: www.sanbi.org.
Printed by: Creda Communications (Pty) Ltd, PO. Box 9403, Johannesburg, 2000.
FINE GCC SPecreneecceccecrter sates ine cry Minton ernie treet = cer MEST cnr are eae ener eee ener ee Vv
LOSSES i ecereaet ncreestoapen Be pet acne iin tots So Cone Ae | A IO Be RRO rein A: Res St Ae oe enn a Vv
NC HIMMEI GP CTINE TICS amraertsce aren ter caper scree Soe RON ate cea y secs esa casa veussaouamerotes oat ee iss nanan tee ssas sass bavepuasowel oantinsaa¥ilsa astiaccuu genet vi
JAE ROUT OS cepenatbeh wrcecetcen ase Teeter Coane cer eEL TOOT RECT eee ee ee ee Vii
JEU, - cette DS ele SS eT) AN (8 cae oo tere incor eee, en Eo ea eee ee een Pet ae vii
INTRODUCTION 1
be MAAIDICTERRLIIS [VAS] [2 10S 0 a OS Re ep ae Re ere ae Sn eer |
2 ages Sot a eg TCG IS gM bl bod ig See acres Oe ie Se ee EE Core Won nck ce Be he Re 2
Db) MAE within tiie’ DIOGIVensity COMSEFVAtlON SECON .c......c.sacenesctscseecesvoreessecsaceceanservecrdvocsersecuscnsecsenssntsetecescavoesediees 3
2 MAW KOUIISSULE IS TVAINIBIENOYO) G13 OT; U0e siteanete pepe tec nse Romar Te aun ey SOP Permernnoh tenon tea ere ere ne err eee 6
SON VaLOULEAN WSEs IS FUAIND BOOKS sits casccctescnscateseccuens exces duces sos sospotensesseptcsictanece usberd auvasuuactshusodi beidaraseardsvees 7
A LOVES TAN D BOOK IS. SiiNUI GM WRED ctecccsosscenescsceccotzssessastacteserorstrscscsczsessereasesestcocsotensodsosoaiestocadussusisestedeeoes 8
Section 1: DEVELOPING AN Maé PLAN 11
imei yak cide: OBJECTIVES @RMREIPROJ ECM 2 | sccsceccecscscccoscevccxrcccsastcesecosecercta seas eecsteoeestiscuteateseeseesuevencsecneoreeecers 12
Pee BOWIE ESE Mite INS ORMAIION COEEECTIED?. si. ccucccecccceccscasccesceccsseetnceerecceecasee ase ceaeetesaseeseteaecheaesecnoare 17
Se HOV ARE THE INDICATORS USED AND! DEVELOPED? ciiicet-cicxcescracccccecesstexececnousnscdscsansecsuscecseoscseasdesasetesnscucoes 20
PVA Mend ODS VIEL BE WUSED TO (GATHER INFORMATION? vacecscccscccoccscecsecesesascaccecceossassssesssnseascosnsoestzcees 28
os) WAM ROUVAYITETEST EAN ROCCE E7610 STN ln 6 Sa eaten oe aA RP BN rt EN o-oo eRo e 30
WACOM al INAV Vile eagtysn ic MIAN Ge tN etna cote crcecenn vec cosecseccectesscieessSostsccustschcosasuccotcueeseas neta oaocre sate tanec ths nouseseesaaseeceaae neneoeede 33
TeV ONY Medes 1c: Si SE UB IAIN ANGE ccs anc csnecscccvncctectcesenscus nats anera sis anatnssasdhcteosisnusnstessicessxceceensazeueedee 35
Pes nOVy Wile tEARNING BE BUIET INTO THE M&E/PROCESS? 2228.8 cts see cccscs. access ansansteectess vas scasssayeeseustencceoxe oy,
Section 2: LEARNING 40
Pel OVy DO ESMEARINIIN Gr IARBEND Herc. cs.tesscstschdaccsecs tcasesstonsstestssesessotsdsetensiees avebeodaastimavicorctntescreaitsalcoeaduveste Mdesnsemeuies 4|
DP aUEARININGHIN EEE PROJECT EAM ses cos sscccsucvesceseetcne-nsccentancesutee-stsaseves eceeseusectbstcnereesetrtctreecucoaccosccnvosrnasttitntaxecess 43
PS eB UIEDINGA LEARNING IENVIROINMENT, | scsszcccqscecctiectecsscscacvecsfecckehccsctecnstusnausctcctnes-dcetnsbucccocccasnsbasasrasuvancevactuccheve 50
Section 3: GATHERING THE INFORMATION 02
Bal eONDERSTANDING YOURCINFORMATION NEEDS” ws icscsccesacecteastescassacueneatossacsevestuzcscntscoatuensrtcatotoutestloteckoccotessueadees 53
Be SELECIINGAND USINGAPPROPRIATIE METHODS 5 sccasce-csoses toetnet <cvceraenseneresnascevaestevdenventrossdtseasticsenadtecvcestectens 57
3.3) MANAGING THE INFORMATION GATHERING SYSTEM i.c...c.cctccccteccaccestcctsencesecercecerevascosescoccstacascatorcorsesesaccceesas 63
Section 4: ANALYSING THE INFORMATION 67
An PESTABLISHING IE THEREARE SIGNIFIGANTT DIFFERENCES 1.....ccccccsccscccsnsecscecssocrercecessientcustesesescuovensensarssiasansass 68
4.2 CLARIFYING AND AGREEING ON THE REASONS FOR ANY DIFFERENCES. o.....esescesessssesteseseseeeneeteneenenes 74
Aa CONDUGTINGIFORMAL EVALUATIONS. © caccsciesessvsnssacsnvscssvensczustesneasversnonevanertrauccsloybaveangoxsbneteatsssbessasGoustceetseneabenani 76
Section 5: ACTING ON THE ANALYSIS 83
Ea DAVY INGO Oil RECOMIMENDATIONS 2 ccsccccscvescrasescovevcusoxssxoss'eqnavtenscusoxvextdrtstunsenhcsrsucarscoseveutestuasiveccveveudavosessescucats 84
Epa ME EE Oa HL Fat SISTINE ANISIIN Gi aces cco cos vepagis se cc c05 casns ca converasucestecdesconesenved Coney na shostexcuechussverdasedysanunanstiskaasakanausaccers 87
Section 6: SHARING THE LEARNING (REPORTING) 93
Bl WAT IS ACCOUNTABILITY? . cccssccvsccessssssescccssosnsusscacnsonsusubessrsstssssneonavsnnrasazensaneansiapsnedses is6:ssi0i/0skteGntceranaeenen GEESE 94
BD REPORTING FORMATS | siccsccsccssocsersercescosronscccnsonavazncucoctpacartecvanesobunstcarenssasenscensercentstenarisonetespensistes traseeS ieee nea nEnEeene enon 99 |
63 FINDING COMMON GROUND: ssccssecccsossiersessssesciscspocosescsntossencanceonessoraseseesossronsorenetebesteneevsnenconeneeteneeneeten a neon ENRORStE 102
)
CONCLUSION 107
FURTHER READING 109
BIBLIOGRAPHY 11
APPENDIX 1: GUIDELINES FOR TRAINERS 113 ;
INTRODUCTION: ciciecicsecnsciccsle ee hlpeacetecsciiaieiesetlicscs BLT een ne 113
Section’ 1: DEVELOPING AN M&E PLAN c..ccccccccsoscossccnssceccovccccaszccesccescosececsversesStieucosssetns nee one 114
Saction 22 LEARNING. .ccccsscéscsacccssccsssonsocsnvenvoxscsssateooncossascasasccucensoseenstersesesosanhecae caps eieseeoas Grete Nie TT Tone I AI7/
Section 3: GATHERING THE INFORMATION. scsccssecccssseccscssscossesecsdssesdessapseecereeste os eelrint eee 120
Section 4: ANALYSING THE INFORMATION .u.ccssssosccsssseccosscssscezesesesessosesssvosetsaestete tee renaey ee ee 123
Section 5:ACTING‘ON THE ANALYSIS): ncccscsicscccscecscconearerocsttceszcuceveececoueets catch ieee eee 125
Section 6: SHARING THE LEARNING csscccccssssscccccecvececcccovesssstacccacnccsscsvgrccusstavses tstoesinv setae 126
eee
We are pleased to present the second in a series of three handbooks that form part of the C.A.PE. Part-
ners Toolbox, following the first publication, Project planning: tools for biodiversity conservation and development
projects, published last year.
We hope that this second publication, focusing on project-level monitoring and evaluation (M&E) and re-
porting, will be of use to organisations and projects across the C.A.PE. partnership.Although some projects
have M&E systems in place, there is always room for improvement in tackling the big questions around the
real impact of our biodiversity conservation work—in terms of both the state of biodiversity and the deliv-
ery of socio-economic benefits.
This resource sets out practical tips for developing an M&E plan, building a learning environment, setting up
an information gathering system, analysing the information, drawing out recommendations and sharing the
learning. | believe it will be an asset to anyone who is designing, implementing or improving an M&E system.
Guidelines for trainers are also provided to enable you to use the material in the book further within your
organisation.
| wish you the best of luck in using this resource and hope it will contribute to the development of excel-
lence in monitoring and evaluation throughout the C.A.PE. programme.
Dipolelo Elford
Chairperson: C.A.P.E. Implementation Committee
June 2008
The Critical Ecosystem Partnership Fund has supported the production of this handbook as a resource
that adds value to our investment in the Cape Floristic Region biodiversity hotspot and to the C.A.PE.
programme.
CEPF believes that it is critical for all of us in the conservation sector to analyse the overall impact of our
work in making progress towards our intended conservation outcomes. This involves setting clear and
coherent objectives and strategies that give our projects focus and direction, an expected results chain or
logical framework, measurable indicators that meet our monitoring and reporting needs, and systematic and
regular processes for collecting and managing data.
All these topics are covered in this handbook, using real case studies that bring the theory of monitoring
and evaluation to life and make it accessible. We welcome the addition of this handbook to the C.A.PE.
Partners Toolbox and encourage readers to apply it to their own projects and programmes.
Nina Marshall
Grant Director
Critical Ecosystem Partnership Fund
June 2008
Acknowledgements
This handbook is an adaptation of the Planning for monitoring and evaluation handbook by Michael Randel,
published by Olive Publications in 2002. Cape Action for People and the Environment (C.A.P.E.) obtained
permission for the adaptation, and commissioned this handbook in 2007-2008.
lt is preceded by and builds on the material contained in the Project planning: tools for biodiversity conservation
and development projects handbook prepared by C.A.P.E. and published by the South African National
Biodiversity Institute in 2007.
In the course of developing this handbook, a number of people made valuable contributions. In the early
stages, the User Reference Group (comprising Tanya Goldman and Onno Huyser) and the Client Reference
Group (comprising Mandy Barnett, Azisa Parker, Caroline Petersen and Monique Damons) provided a clear
foundation for the development of the handbook.
Project managers from three projects and a small grants funding agency generously provided time and
insight for the purpose of generating case study material. They are:
* Cape Flats Nature.
* The C.A.PE. Conservation Stewardship Programme, located in CapeNature.
* Greater Cederberg Biodiversity Corridor.
* Table Mountain Fund.
Detailed information about these projects can be found in Fynbos fynmense: people making biodiversity work
(Ashwell et al. 2006, SANBI Biodiversity Series 4, published by the South African National Biodiversity Institute,
Pretoria) or on the C.A.PE. website, http://www.capeaction.org.za.
Participants in the Project Developers’ Forum held on 18 and 19 March 2008 also contributed material used
in the examples in the handbook and in further refining the content.
The C.A.P.E. Co-ordination Unit oversaw the process of producing the material for the handbook. Material
was developed and compiled by Carol-Ann Foulis and Jenny Whitehead. Sue Soal skilfully facilitated the
Project Developers’ Forum and helped with the conceptualisation of the handbook.Anne Kroon brought
her creativity and experience to bear in helping to develop the Trainer Guidelines. Judy Norton meticulously
edited the handbook. Michael Randel, as the original author of the Planning for monitoring and evaluation
handbook, provided in-depth and valuable comments on a draft version of this handbook.
SANBI’s Publication Unit was responsible for technical editing, design and layout, and cover design for the
handbook.
Financial support was provided by the Critical Ecosystems Partnership Fund and the Global Environment
Facility through the C.A.PE. Biodiversity Conservation and Sustainable Development project.
The C.A.P.E. Programme is hosted by the South African National Biodiversity Institute and is supported by
24 signatory partners.
vi
AAR After Action Review
C.A.P.E. Cape Action for People and the Environment
CDRA Community Development Resource Association
CEPF Critical Ecosystems Partnership Fund
CFN Cape Flats Nature
CFR Cape Floristic Region
CNC Cape Nature Conservation
EIA Environmental Impact Assessment
GIS Geographic Information System
LFA Logical Framework Approach
M&E Monitoring and Evaluation
NGO Non-government Organisation
OvI Objectively Verifiable Indicator
PCM Project Cycle Management
PLA Participatory Learning and Action
PME Participatory Monitoring and Evaluation
PRA Participatory Rural Appraisal
SA South Africa
SANBI South African National Biodiversity Institute
SCM Sustainable Conservation Management
TMF Table Mountain Fund
TOR Terms Of Reference
WWE World Wide Fund for Nature
Adaptive management Management that is responsive to learning.
Action learning A learning process that, by reflecting on past experiences, draws out relevant lessons
and supports the preparation for future actions.
Activities The steps that the project must take to achieve the Outputs.
Assumptions External factors, not under control of the project, that are identified as being necessary for
project success; part of the project plan (LFA).
Beneficiaries The people who will experience improved conditions (benefits) as a consequence of the
project targeting their needs.
Benefits The positive conditions of change resulting from a project.
Development goal Describes the benefits that will result from the project.
Development objective See Development goal.
Enabling factors See Assumptions.
Ex-ante evaluation An assessment of the project plan and approach, before it is implemented, to assess
whether it has a good chance of achieving its objectives.
External factors See Assumptions.
Vil
Formative evaluation A periodic review of the project that allows continuous feedback to inform
ongoing changes and improvements in a service or a product.
Immediate objective See Project purpose.
Impact The positive differences resulting from the project; often seen in the benefits for specific groups.
Impact evaluation An external and in-depth study of the impact of a project on its beneficiaries; usually
carried out 3—6 years after the project has closed.
Indicator The measure that is used to assess if an objective has been achieved, or what progress has been
made.
Inputs The human, financial and material resources required to implement the project.
Intervention A deliberate action on the part of the project to influence change in a social system.
Logical Framework Approach (LFA) A method for project planning that focuses on objectives or
outcomes.
Means of verification The location or source of the evidence used as indicators of project
achievements.
Mid-term evaluation An external assessment of the project, usually conducted midway through the life
of the project, that focuses on its performance and assesses whether the objectives continue to be relevant.
Objectives Describe what we are trying to achieve. There are four levels of objectives in the LFA
approach: Goal, Purpose, Outputs and Activities.
Objectives analysis A tool to visualise an improved future, linked on a,‘means-end’ basis.
Outcomes The positive changes that result from the project’s intervention; achieved by the use of the
outputs of the project.
Outputs Describe the responsibilities of the project, and the services and products it will deliver.
Participatory Monitoring and Evaluation (PME) An approach to monitoring and evaluation that
emphasises the role and views of the target groups and beneficiaries of the project.
Project A set of planned activities designed to achieve specific objectives with given resources within a
specific time frame.
Project Cycle Management (PCM) A process of managing a project from the start of the
preparation and planning phase, through to implementation and evaluation.
Project purpose Describes what the target group will be able to do because of the support they receive
from the project.
Results See Outputs.
Summative evaluation Evaluation that takes place to review the accomplishments of the project and
to inform high-level decisions of future funding and policy.
Sustainability The ability of something to maintain its value over a long period, without external support.
Target group The people whom the project aims to support, and whose consequent actions will bring
about a benefit to themselves and/or others.
Terminal evaluation An evaluation at the close of the project on whether it has accomplished its
objectives or not.
Vili
;
Monitoring and evaluation: tools for biodiversity conservation and development projects I
INTRODUCTION
1. WHY THIS HANDBOOK?
Most people working in the biodiversity conservation sector have their hands full with tasks and
deadlines associated with project implementation. This handbook focuses on a particular set of
activities that are integral to the process of implementation, but that are also distinct from it—
namely the activities that make up monitoring and evaluation (M&E). These activities have a special
quality—they are about observing and gathering data, about reflecting and learning. Through this
process, M&E has an important contribution to make to the success of projects. It can help to cre-
ate ones that are innovative and pioneering, and impart a different way of thinking, doing and relat-
ing.Another way of describing M&E is that it is ‘learning while doing’.You do not stop implementa-
tion to ‘do’ M&E. It is part of the process of implementation.
This handbook is the second in the C.A.P.E. Partners Toolbox series. The first handbook focused
on project planning and drew on the Logical Framework Approach (LFA) as the basis for managing
the planning and implementation of projects. LFA terms and concepts form the foundation for this
handbook on M&E. Together, these handbooks are part of building a shared language about project
management within C.A.P.E.We hope this will enable project partners to talk to one another, that
it will fuel discussion and debate, raise questions as well as provide useful tools and material. There
is no one way of doing M&E (or planning). The intention is to help you become more confident and
familiar with M&E so that you can adapt and change the tools and methods to suit your context.
The focus of this handbook is on project monitoring and evaluation and what is associated with
it: clear objectives, a particular form of support or intervention, a set time frame, a defined target
group and beneficiaries.
The notion of target group is central to projects—who is it that the project aims to support? What
are the new actions that are required to bring about a benefit to the environment, to themselves
and/or to others? Beneficiaries are also important—who or what stands to benefit from these
changes? Within the context of biodiversity conservation, both the environment (or biodiversity)
and people can be viewed as beneficiaries.
The C.A.PE. programme is characterised by a diversity of projects at different stages of organi-
sational development and project implementation. The scale and complexity of these projects is
wide ranging, as are the resources that are available to them.What is common across the projects
is a growing interest in developing M&E capacity and building an M&E ‘practice’ that is identifiable
within the C.A.P.E. programme.
2 Monitoring and evaluation: tools for biodiversity conservation and development projects
TERMINOLOGY/AT/A\GLANCE: PRACTICE
There are a number of characteristics of a practice. It comprises a set of activities in which people
engage on a regular basis, and the behaviours that accompany these activities. More specifically, it is
work (or activities) that takes place within a particular discipline with its own body of knowledge, ways
of doing things and code of conduct that have been developed and tested over time, e.g. conservation
practice. For this reason, it has the power to hold its practitioners to account. However, to maintain the
vitality of a practice, practitioners are constantly contributing to it and changing it along the way. Lastly, a
practice (in whatever field) is recognisable to others working in the same field.
Thus the notion of ‘practice’ can be applied to the field of M&E, and practitioners working within it.
(Collingwood 2007)
This handbook aims to provide you with:
* An overview of some of the key issues in project M&E.
* A guiding framework within which you can develop your M&E plans.
r
i
i
1
1
1
1
'
" Tools, concepts and exercises to build your own M&E practice.
1
=
The handbook will focus largely on the monitoring and evaluation activities carried out by project
staff.
a) Recent trends in Mae
The world of M&E has become increasingly specialised terrain, partly to address the complexity
and sophistication of development projects more adequately. This can sometimes have the effect of
limiting who can and cannot do M&E. There are, however, some new and exciting trends that firmly
locate M&E in the hands of those who are responsible for project management and implementa-
tion. We will look at two trends.
The first is a growing recognition amongst the various development players that a key aim of M&E
processes is that of learning. If people, projects and organisations do not learn from their involve-
ment in and contribution to M&E activities, then it has failed in one of its core objectives.
The effect of this has been to bring learning to the fore in newer models of M&E, by placing greater
emphasis on the importance of reflecting on the lessons of implementation and learning from
them. This indicates a shift away from upward accountability as the driving force in the rationale for
project M&E.‘Accountability’ remains a key issue. However, this concept has been extended to that
of multiple accountability, focusing on horizontal and downward accountability as well.
A recent nine-month study, commissioned by the Ministry of Foreign Affairs of the Netherlands,
looked at the issue of M&E in its theme-based co-financing programme. It had this to say about
where M&E has come from:
Monitoring and evaluation: tools for biodiversity conservation and development projects 3
‘Historically, accountability orientated M&E has often focused primarily on upward accountability for the
expenditure of funds and the delivery of outputs.Attention to outcomes and impacts and how feedback
mechanisms could be used to help those implementing programmes improve their performance has been
minimal. (De Ruijter et al. 2006: 9)
Many donors and development organisations are now grappling with how to re-think M&E to make
it more relevant, and to increase its contribution to the impact of development projects, and to the
sustainability of these impacts.
The second positive trend is to view monitoring and evaluation as two sides of the same coin—‘as
an integrated process of continual gathering and assessing information to make judgements about
progress towards particular goals and objectives, as well as to identify unintended positive or nega-
tive consequences of action’ (De Ruijter et al. 2006: 10). In this scenario, project managers and im-
plementers actively value their work and assess its contribution to development goals and project
outcomes. The alternative is to lose the meaning of their work:
“The separation of monitoring from evaluation has been partly driven by the classical approach to develop-
ment projects, in which evaluation was undertaken every now and then by external experts, while monitor-
ing was the task of project implementers. It is exactly this scenario that has resulted in an inability of many
development initiatives to learn effectively as it disconnects the information collection from the sense-mak-
ing that precedes improved action’ (De Ruijter et al. 2006: 10).
Even the separation of ‘planning’ from ‘M&E’ can set up an unhelpful distinction between two proc-
esses which, in practice, are intimately located and held together under the umbrella of project
management.
These shifts within the field of M&E challenge us, as development practitioners, to think more
deeply about our practice and to find ways to articulate and describe this to others. This means
having a clear ‘approach’ to the work we do—a way of describing what we do and how we do it,
based on reality (Soal 2001).Wéithout this frame of reference, it becomes difficult to talk about
development work and its impact both amongst ourselves (people working within projects) and to
donors. Two questions become important:
‘Do we know what is valuable about our work?’
and
‘Do we know how to go about valuing our work?’
This means that any M&E framework has to begin by asking, ‘What is our work?’ and more specifi-
cally, ‘What is our practice?’. This requires an openness to one of the building blocks of M&E proc-
esses and systems: reflection. We have to reflect in some way or form, individually or collectively,
formally or informally, on what we are doing.
Much of the recent literature on evaluation therefore refers to ‘practice improvement’ and the role
that evaluation can play in contributing to it.
b) Mae within the biodiversity conservation sector
The task of conserving the rich biodiversity of the Cape Floristic Region (CFR) is a large and sig-
nificant one, as is C.A.PE’s contribution to it.
4 Monitoring and evaluation: tools for biodiversity conservation and development projects
Working in this field, we must be able to demonstrate progress—go beyond implementation to
show results and effects. This requires understanding and working in an increasingly complex ter-
rain that combines ecological, social, political and economic factors.
In M&E in biodiversity conservation, many of the objectives are quantifiable. Perhaps this is not good be-
cause it can mean that we do not spend enough time thinking. Maybe we have to look at this?
(Conservation Partnerships Programme)
Researchers looking at the usefulness of M&E within the conservation sector argue that (1) good
project management is closely linked to effective M&E systems and (2) good M&E contributes to
better decision-making and therefore enhances the success of biodiversity conservation (Stem et al.
2005).
Stem et al. (2005) propose four (sometimes overlapping) reasons for undertaking M&E within the
conservation sector:
* For basic research purposes—to gather or generate knowledge about a subject to understand
it better.
= For accounting and certification purposes—to assess the extent to which a programme or
project is fulfilling its obligations to donors, the public, government or other enforcement enti-
ties.
= For status assessment—to assess the condition or status of a particular conservation entity
(species, population, ecosystem) at a particular moment. This is generally irrespective of a spe-
cific intervention designed to affect the variable. Tools associated with status assessment include
population monitoring, rapid assessments, state-of-the-environment monitoring, and report
cards and scorecards. This kind of monitoring is often indicator driven.
= For effectiveness measurement—to measure effectiveness of discrete interventions employed
by specific actors. These can be divided into two categories: impact assessment and adaptive
management.
This handbook tackles issues of M&E that largely fall under ‘effectiveness measurement’. In draw-
ing on the LFA and project approach to M&E, the handbook links with the notion of ‘interventions
employed by specific actors’ (Stem et al. 2005). You may also find that some of the principles in this
handbook dovetail with those of ‘adaptive management’.
TERMINOLOGY/AT/AGLANCE: ADAPTIVE MANAGEMENT;
According to Wikipedia, adaptive management originated in the 1970s out of the work of two ecologists,
Holling and Walters, at the University of British Colombia, Canada. It is defined in the following way:
Adaptive Management (AM), also known as Adaptive Resource Management
(ARM), is a structured, iterative process of optimal decision-making in the face of uncertainty, aiming to
reduce uncertainty over time via system monitoring. In this way, decision-making simultaneously max-
imises one or more resource objectives and, either passively or actively, accrues information needed to
improve future management.AM is often characterised as ‘learning by doing’.
Monitoring and evaluation: tools for biodiversity conservation and development projects 5
There are a number of features associated with this approach:
= Iterative decision-making (evaluating results and adjusting actions on the basis of what has been
learned).
= Feedback between monitoring and decisions (learning).
= Embracing risk and uncertainty as a way of building understanding.
Using Bayesian inference—formulating a hypothesis and then collecting evidence that supports or
refutes it.
(http://en.wikipedia.org/wiki/Adaptive_management)
=
Adaptive management is like...dancing on the back of a truck while it is moving. You have to be fit and agile! \G
(Project Developers’ Forum, |8—19 March 2008)
6 Monitoring and evaluation: tools for biodiversity conservation and development projects
2. WHO IS THIS HANDBOOK FOR?
There are two audiences in particular.
e Project developers, managers and implementers
These are people who are leading and managing the development and implementation of
projects and are responsible for thinking about how to do M&E for their particular project.
Most often, they are called the ‘project manager’ but they may also be project staff who are
leading a particular aspect of the project.
Because the handbook presents an overview of an M&E system, it is useful for project
managers and project staff who are new to their positions and are looking for ways of ‘getting
started’. There are also sections in the handbook that are more challenging and that will extend
project managers and project staff who have some experience of M&E.
What characterises this audience is that members are self-motivated to learn and to develop
their skills and understanding of M&E.
e Intermediaries
These are people who play a supporting role to projects and project staff. They can broadly
be categorised as ‘capacity builders’. What characterises this audience is the role they play
in supporting the learning of others. In some cases, they may be the project manager who is
located within the project. In other cases, they may be external to the project, such as the
project developer.
Their purpose in using this handbook would be to develop and deepen their own
understanding and skills in M&E so that they can build the capacity of their colleagues. The
design of the handbook takes this purpose into account as well.
We have made certain assumptions about you as a reader:
* You have prior working experience and some experience in biodiversity conservation
projects.
= You have tertiary-level qualifications and are comfortable engaging with written texts.
* You occupy positions where you are willing and able to apply the learnings and have the
opportunity to do so.
We have also assumed that the audience comprises a mix of scientists and social scientists,
and that part of the purpose of this handbook is to build a common language for M&E.
Monitoring and evaluation: tools for biodiversity conservation and development projects 7
3. HOW YOU CAN USE THIS HANDBOOK
You can use this handbook in a number of different ways.
e Proactive learning
You might be reading this handbook, alone or as part of a reading group with your peers, to
learn more about project monitoring and evaluation.
e In response to a problem
You may recently have had a problem in your work, and are concerned that your existing M&E
processes failed to give you advance warning of this problem. You might be looking for ways to
improve your project’s ability to monitor and learn from experience.
e In response to an opportunity
You may be in the midst of preparing a new project, and are considering how to include M&E
concepts and tools into the work. You may be interested in practical ideas that can be part of
your approach from the beginning.
Depending on your need and interests, you may choose to use this handbook in a number of
different ways. You may have the time to read it from cover to cover, or you may dip into it
from time to time, finding ideas that will help you deal with your particular questions. We would
encourage you to read it in the sequence in which it is presented.
This handbook can also be used as a resource by your organisation or project team. For
example, parts can stimulate discussions on your approach to project monitoring and
evaluation.
Monitoring and evaluation: tools for biodiversity conservation and development projects
4. HOW THE HANDBOOK IS STRUCTURED
The Monitoring and evaluation: a pathway to learning handbook will take you through six steps that
outline an approach to project monitoring and evaluation. We will present the objectives for
each section, some examples from the field, relevant ideas and tools to stimulate your thinking.
There will also be exercises for you to complete.At the end of each section, we provide a list of
References and further reading.
&
Developing an M&E plan
Thinking through the demands that are likely to be made on your M&E system is the first step.
Through thinking about what kind of system you need, you can develop a framework that brings
it into being. You will explore a number of important issues that will influence your approach to
project M&E.
Learning
Learning forms the background within which M&E happens. By understanding some of the key
concepts and processes associated with learning, you can start to create the environment that
will encourage and facilitate learning.
Gathering the information
Information is the fuel that drives the M&E system. You will have to collect information on an
ongoing basis. The indicators for your work will tell you what to look for. There are a large
number of methods you can use to collect both quantitative and qualitative information.
Depending on your information needs, you might be looking for primary information (new
information that you are the first to collect) or secondary information (information already
collected by other groups or organisations that is nevertheless useful to you).
Analysing the information
You will have to make sense of the information that you have collected. This means exploring
what happened and why it happened as it did, looking for reasons that helped or blocked the
achievement of your objectives.At this stage, you could conduct a formal evaluation to deepen
this analysis further.
Acting on the analysis
The whole M&E process has been building towards this step. By drawing on the information
you have collected, and the insights you have developed, you can now decide what follow-up
action should take place. This often takes the form of recommendations. It also requires effective
decision-making that is supported by the various stakeholders.
Sharing the learning
Lastly, as part of your accountability, you will have to document what you have noticed and
plan on changing, and communicate these findings to project stakeholders. If you have learnt
something significant from your experiences, you may want to think about how you can share
this with interested people in your own and other organisations. Reporting to funders is also a
key activity in this step.
Monitoring and evaluation: tools for biodiversity conservation and development projects 9
Things to look out for
To illustrate the application of various ideas and tools from the field of M&E, we have used examples from
three C.A.PE. projects throughout the handbook. We have described these projects in generic terms, i.e. Urban
Sustainable Conservation Management Project, Conservation Partnerships Programme, Biodiversity Corridors
Project and Small Grants Funder.
Resources
This part of the handbook provides you with tools and ideas to help you with the practical ‘how-to’
aspects of project M&E. Some of this material is presented as categories, concepts and guidelines.
Exercises
We have used exercises to encourage you to reflect on your current thinking and practice in relation to
M&E. These exercises provide you with an opportunity to make notes about any new ideas or thoughts
you have about M&E and how you apply them in your project.
REFERENCES AND FURTHER READING
COLLINGWOOD, C. 2007. Revealing practice, re-imagining purpose, claiming our place. Biennial Practice Conference:
McGregor, May 2007. http://www.biennial.cdra.org.za (accessed January 2008).
DE RUITJER,A., DIETZ,T..VAN GONGEN, E., HELMSING, B. & KNORRINGA, P. 2006. Evaluation of the Theme-based
Co-financing Programme. Cross-cutting study: monitoring and evaluation. Prepared for Ministry of Foreign Affairs of
the Netherlands. http://www.tmf-evaluatie.nl/eindrapportage%20M&E.doc (accessed January 2008).
SOAL, S. 2001. How do we know what difference we are making? Reflections on measuring development in South
Africa. Adapted from a presentation to a workshop between Christian Aid and South African partners, Durban,
October 2001. http://www.cdra.org.za (accessed January 2008).
STEM, C., MARGOLUIS, R., SALAFSKY, N. & BROWN, M. 2005. Monitoring and evaluation in conservation: a review of
trends and approaches in conservation biology. Blackwell, Gainesville, Florida.
Monitoring and evaluation: tools for biodiversity conservation and development projects 11
Section 1: DEVELOPING AN Mae PLAN
The M&E plan provides you with an overall framework for your M&E activities. Its purpose is to
remind you of what you are trying to achieve through the project, and help you to think through
your information needs to effectively monitor, evaluate and manage your project.
The purpose of planning is the process, not the artefact.
(Small Grants Funder, citing Eisenhower, 1890-1969)
We have identified eight key questions that will help lay the foundation for a good M&E plan:
|. What are the objectives of the project?
Who will use the information collected?
How are indicators used and developed?
What methods will be used to gather information?
Who will participate in M&E?
When will M&E take place?
How will the M&E system be managed?
i ee er)
How will learning be built into the process of M&E?
, By the end of this section, you will have:
" Explored eight questions that will assist you in developing or revising your own M&E plan.
Been reminded of the links between M&E and project planning.
Reflected on what you are currently doing and why you are doing it.
A better understanding of the role of objectives and indicators in project M&E.
;
1
1
1
i
1
1
1
1
1
1
1
Deepened your skills and understanding of how to develop indicators. ,
1
al
The devil's in the detail
It took us a couple of months to develop our logframe.We had an extensive participatory process, which meant
we wrote and re-wrote the logframe. Because our project was about mainstreaming biodiversity, there were a
variety of voices that we had to listen to. It was furthermore a high-risk project, which made the funder nervous.
This meant that we had to be more specific. We sometimes felt as if we were in a ‘catch 22’ situation with the
detail. Initially, | was holding people very tightly to the logframe. But in the end, it allowed us to achieve huge
things very quickly.
(Urban Sustainable Conservation Management Project)
ZS
«
SZ
«
12 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.1 WHAT ARE THE OBJECTIVES OF THE PROJECT?
An important starting place for any M&E plan are the goals and objectives of the project. If they are
clear and specific, then you can be clear about what you are trying to monitor and evaluate.
Behind this simple statement lies the experience of trying to monitor and evaluate projects that
do not have clear objectives for the work they are doing. In such cases, it is difficult to know what
information to collect.As a consequence, it is almost impossible to assess whether or not the
project has succeeded.
While some projects know what they want to do, they are not always clear about what they will
achieve through this work. For this reason, planning methods like the Logical Framework Approach
(LFA) and Results-based Management have been developed. These methods place a great deal of
emphasis on first clarifying the desired destination (the Goal and Purpose) before you work out
how you will reach it (through the Activities and Outputs).
This requirement of clear plans should not stop you from experimentation and piloting. It is still
possible to try out new ways of working and to have a plan that is clear about this.
Different types of objectives
The Logical Framework Approach (LFA) method (as introduced in the C.A.PE. Project planning
handbook) outlines four main levels of objectives. They are linked in a hierarchy. These different lev-
els are important for project M&E as they help distinguish between a project’s outputs (the work
carried out by staff) and its outcomes (the changes that result from the project).
Often more time is spent on monitoring outputs because it is within the control of the project and
therefore easier to measure on an ongoing basis.As a project manager, you are able to see what
goods, services and products the project is making available to the target group. It is much harder
to measure what the target group is doing with your goods, services and products, or to measure
the longer-term impact it has on the broader group of beneficiaries, or on the environment. Yet it is
important to the success of a project to assess the outcomes, as it is the only way to know wheth-
er all your work (the outputs) is effective.
TERMINOLOGY/AT/A(GLANCE: OUTCOMES
The Development Goal and Project Purpose are often jointly referred to as
Outcomes This is the approach that will be used in this handbook.
Monitoring and evaluation: tools for biodiversity conservation and development projects 13
Here is a simple description of these four levels of objectives:
This describes the longer-term benefits to which the project will contribute,
either through a change in the environment and/or through its contribution to
Development the lives of beneficiaries. The Development Goal normally has a clear link to the
Goal initial problems that the project was set up to address. There is normally only
one Development Goal in the project plan. Other projects may be making a
different contribution to the same Development Goal.
This expresses the actions that the target group will take to bring about the
desired change (identified in the Development Goal). The Project Purpose
Project Purpose often describes a change in the behaviour of the target group, or changes in the
situation in which they are located. It is unusual to have more than two (or at
most three) Project Purposes in the plan.
These are the goods, services and products that the project makes available
Outputs to the target group.A manageable project will normally have between five and
eight Outputs.
The various tasks that the project staff carry out to deliver the Outputs to the
Activities target group are called the Activities. Each Activity is clearly linked to a specific
Output.
The term ‘target group’ is used here to indicate the specific groups and organisations that the
project is trying to support. This recognises that a project often cannot assist everyone directly,
but can work with and through others (the target group) to help initiate and support change in a
wider group (the beneficiaries). Within the context of biodiversity, the notion of ‘target group’ is
significant in bringing to the fore the attitudes, values and behaviours of people that have to change
in order to achieve the objectives of biodiversity conservation.
Project M&E is also interested in two additional areas that are often found in project plans.
These are the resources that are needed to implement the project. Inputs will
Inputs include financial and human resources, and physical resources such as land.
They may also include contributions from projects in which there are multiple
partners.
The Assumptions provide a way to identify significant enabling factors in the
Assumptions project’s external context that might influence project success. By making these
factors clear and explicit, it will be much easier to monitor whether they are
affecting the project's efforts.
The focus of this handbook is on monitoring and evaluating objectives at the level of development
goal, project purpose and outputs. We will also look at monitoring assumptions (which are
sometimes referred to as enabling factors).
14 Monitoring and evaluation: tools for biodiversity conservation and development projects
Monitoring the achievement of objectives at the level of activities and inputs will be addressed in a
third handbook on Project implementation.
Ss Example:
Assumptions
|. Government is committed to
The unique biodiversity in the urban lowland
fragments of the Cape Floristic Region is
conserved in a way that benefits people of
biodiversity conservation at
high levels and line agencies
are required to comply with
directives to incorporate
biodiversity into planning and
to collaborate through the
mechanisms established.
: . Government institutions will
Development the Cape Flats and is embraced as a valuable ;
otal ae ; have the capacity to integrate
Goal element of urban life in line with the City .
- and implement forward plans
of Cape Town’s Integrated Metropolitan
; eneguty ae efficiently.
Environmental Policy and Biodiversity a ne
. Communities willing to
Strategy. Sed
engage in opportunities
developed in protected
areas and sufficient capable
civil society organisations
are available and willing to
undertake C.A.PE. activities.
. Funds are raised to support
: continued employment of
Sustainable conservation management :
AS. a ; ; the urban conservation
y of sites in the City of Cape Town’s Maen EU
Project biodiversity network is achieved through =
: : operational costs into the
purpose active partnerships between government, nine
the private sector and community-based
Leeda Reasonably low turnover in
organisations. ‘ :
community leadership allows
capacity to be built.
|. Sustainable conservation management is
demonstrated at four pilot sites within
the City of Cape Town’s biodiversity
Outputs conservation network: Edith Stephens
Wetland Park, Harmony Flats Nature
Reserve, Macassar Dunes and Wolfgat
Nature Reserve.
2. A new cohort of skilled urban
conservation managers and champions
from surrounding townships is
established to conserve the biodiversity
of the Cape Flats, and is supported by
Cape Flats Nature to implement all other
outputs.
Monitoring and evaluation: tools for biodiversity conservation and development projects
. A campaign is conducted to gain broad
support for conservation of the four pilot
sites and the work of Cape Flats Nature
through awareness-raising, promoting
use of the sites by the people of the
Cape Flats, and securing benefits from
biodiversity conservation for the local
communities surrounding the pilot sites.
. An advocacy campaign is conducted
to secure support for biodiversity
conservation on the Cape Flats at
all levels of government and lobby,
particularly local government leadership,
around specific issues as necessary.
. Lessons for sustainable urban
conservation management practice
from the work of Cape Flats Nature at
four pilot sites are captured and shared,
and approach of Cape Flats Nature is
introduced at two additional sites in
the City within the context of a roll-
out strategy for the City’s biodiversity
network.
16 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Reviewing your objectives
Revisit your plan for your project. Spend 10 minutes reading it through. Consider the following ques-
tions:
e Do you have objectives for each of the levels of the project?
The project plan should identify both what you will do (outputs) and what you will accomplish
(development goal and project purpose). If you do not have objectives at these different levels, there
may be a problem in the design of your change intervention. This could be a risk to your success.
Are you clear on what your Assumptions are?
You do not want a long list of the many factors that might influence your project. But you should be
able to identify a small number that you believe could have a significant influence on project success,
and that you are not sure will actually occur. These are the Assumptions that you should monitor.
Are your objectives realistic?
You should feel confident about being able to accomplish the objectives with the resources available
to you.
Taking the external context into account, the objectives should not be too ambitious in terms of:
How much you can accomplish (quantity).
The standards to which you will be able to do this (quality).
The time within which you will achieve these goals (time).
The size of the geographic area within which you will work (location).
The various target groups who will benefit from your project (people).
Will you be able to measure the objectives?
To collect information on your progress, you will have to be able to measure each of the objectives.
While tangible and concrete achievements can often be assessed, it can be more difficult to measure
for qualitative components. This issue is discussed in Section 3.
Do you have clear timeframes for the objectives?
You will need realistic deadlines and timeframes for accomplishing the objectives. For those objec-
tives that will take some time to be completed, you may have to establish some intermediate tar-
gets. These intermediate targets should state what you will have accomplished at particular periods,
such as at the end of each six-month period.
Monitoring and evaluation: tools for biodiversity conservation and development projects 17
1.2 WHO WILL USE THE INFORMATION COLLECTED?
Who will use the information collected through your M&E efforts? This is a very important ques-
tion as it will shape your approach to M&E in significant ways. Knowing who your end-users are is
important for the following reasons:
e You can clarify the objectives of your M&E work
As you are likely to have different audiences, you may be aware that they often have different
needs and interests. You might compile certain information for your own project, but you may
have to present the same information in quite different ways to your stakeholder groups. Ask:
« Who wants to know about what we are doing?
# What do they want from us for their interest to be satisfied?
e You can focus your M&E efforts on appropriate information and issues
By being clear about who will be served by your M&E work, you can ensure that you collect
appropriate information to meet these needs. This can also save you from wasting time and
resources collecting information that is of no interest to any of the end-users.
A further benefit is that you can establish when this information will have to be presented to
the end-users. Ask:
# What information do they want?
# When do they want this information?
There are potentially a large number of end-users for information about your project, and each of
them may use the information in quite different ways:
* Beneficiaries and/ or the target group may use the information to understand the
changes they are experiencing and are part of; to understand the reasons behind decisions
that affect the support that is provided to them; and to contribute to their ownership of the
project, which will improve the chances of sustaining the effectiveness of the project.
" Project staff can use the results to deepen their understanding of the context of their
work; to discover how well they are implementing activities; to find out how the effects of
their actions are perceived by others; to understand the reasons for management decision-
making; and to learn what problems have to be solved.
« Project and organisational management may use the results to discover reasons for
problems; to make decisions for planning and implementing the project; to use the lessons in
planning other projects; and for policy-making.
= Donors may use the information to satisfy accountability and reporting requirements; to
decide whether or not to continue funding the work; to learn about the effectiveness of
different types of projects; and to learn lessons that will influence their support for develop-
ment occurring in your sector or country.
There may be other stakeholders that you want to give attention to such as government
departments, collegial interest groups and peers, or the general public.
18 Monitoring and evaluation: tools for biodiversity conservation and development projects
TERMINOLOGY/AT/A\GUANCE:
It is common practice to think about accountability only as upwards—to your donors and funders, and
to your board of directors. However, accountability can be extended in a number of directions:
» Downward accountability—to those to whom you provide services or who you say you
represent.
« Horizontal accountability—to peers and fellow professionals (conservationists)—with
a view to meeting certain shared values and standards that are in the interests of the reputation of
the sector as a whole.
= Inward accountability—to colleagues, and to the stated purpose and values of the project/
organisation.
To the extent that individuals see themselves as accountable in a variety of ways, this encourages greater
self-management and, ironically, allows for more freedom rather than control.
(Cavill & Sohail 2007)
Ss A broad range of information needs...
‘A video will be produced as a marketing tool to promote stewardship throughout the CFR.’ (the beneficiaries)
‘Annual audits are carried out with the farmers who are participating in the programme. There could be more
communication with them, on an ongoing basis, about their needs, and perceptions of the benefits of the
programme.’ (the target group)
‘Reporting provides information to the funders. Others are kept informed through the database. Everyone feeds
information into it and has access to it, so they don’t need reports.’ (project staff and management)
‘Extension staff type the information into the database. The GIS technician checks it and then it is fed through to
head office.’ (project staff)
‘The funder’s reporting requirements are programmed into the M&E system. CapeNature’s Scientific Services
unit collates the data and produces the required quarterly and annual reports.We also need to provide
information required by the host programme (donor and host programme).’
(Conservation Partnerships Programme)
Meeting the needs of all potential end-users may be difficult as it could demand more resources
than you can make available. The needs of end-users may also conflict with one another.As you
consider the priorities and objectives of your M&E framework, you may have to choose whom to
focus on at various points in the project cycle. You might also consider negotiating with users who
have some similar interests, to see if you can propose an M&E product that can satisfy their need
without meeting all their requirements. Donors and others are willing to be flexible once they
understand the cost and burden of reporting, and see that you still wish to be accountable and in a
good relationship with them.
The purpose of thinking about the needs of the various end-users is to help you set appropriate
objectives for the project monitoring work.
Monitoring and evaluation: tools for biodiversity conservation and development projects 19
Exercise: Clarifying the information needs of project staff
Project staff both generate and use information about the project. It is important to clarify the needs
and demands on them, and their expectations in relation to information. Consider their needs in terms
of learning, decision-making and accountability.
The following questions may help:
What kinds of information do project staff need?
Where will they get this information from?
How often, and when, is this information required?
In what format should the information be made available to project staff?
How will they use the information?
20 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.3 HOW ARE INDICATORS USED AND DEVELOPED?
Indicators tell you what you have to measure to assess whether the project is achieving its
objectives. They can range from hard facts and numbers, to changes in people’s behaviour, actions
and attitudes.
If you are on a car journey, the road signs allow you to see how close you are to your destination (feedback
from the external environment). The odometer informs you how far you have travelled since the start of
the journey, while the petrol gauge tells you whether or not you will have sufficient resources to reach your
destination (feedback from internal sources). The information can be compared with your plans: ‘After
driving for three hours, | expect to have travelled 300 km and to still be 50 km from my destination. |
should still have a quarter tank of petrol remaining, enough to reach the destination.’
(Randel 2002)
In a similar way, the M&E process provides both internal and external feedback to project staff.
TERMINOLOGY/AT/A\GUANCE:
According to the WWF (2005), an indicator is ‘a measurable entity related to a specific information
need, such as the status of a target, change in a threat, or progress toward an objective. Indicators can
be quantitative measures or qualitative observations. Good indicators meet the following criteria:
Measurable: Able to be recorded and analysed in quantitative or in discrete qualitative terms.
Precise: Presented or described in such a way that its meaning will be the same to all people.
Consistent: Not changing over time so that the same phenomenon can be measured over time; for
example a currency that inflates or deflates in value is not a consistent measure of wealth.
Sensitive: Changing proportionately in response to actual changes in the condition or item being
measured.
Quantitative and qualitative indicators
A hot topic within the world of M&E is the difference between quantitative and qualitative
objectives and indicators. Particularly in a sector which is steeped in the sciences, the tendency is
to place emphasis on what can be counted and quantified. This is made attractive by the many tools
that are available for these purposes, for example in sampling and statistical modelling.
An important factor to bear in mind is that quantitative data are based upon qualitative judgments
and qualitative data can be described and manipulated numerically (Trochim 2006). Thus, for
example, the selection of a particular method for sampling as opposed to another method is a
judgment in itself. Similarly, the range of attitudes towards prescribed burning in fynbos areas can
be quantified, e.g. the number of landowners in favour and the number of people against. Thus the
qualitative and quantitative aspects of M&E are not as polarised as you might have initially thought.
There are, however, some helpful distinctions:
Monitoring and evaluation: tools for biodiversity conservation and development projects 21
e Quantitative indicators can be measured through direct observation
Four concise briefing documents about biodiversity conservation on the Cape Flats and its benefits for
townships, are developed and distributed to politicians and officials every quarter.
e Qualitative elements may not be easy to observe
You may have to develop specific indicators to assess quality elements.
Did the politicians and officials understand the briefing documents and incorporate them into their work? ( i
e Other elements may be difficult to assess due to technical challenges or
the expense of measuring them
You may have to use indirect or proxy (‘in place of’) indicators. Look for indirect measures that
will be appropriate in the context of your project.
If there is a clear link between the proxy indicator and the objective you are trying to measure,
then you should be able to make informed judgments about the status of the objective.
Is there any reference to biodiversity conservation on the Cape Flats in the development plans on which the
targeted officials have worked?
Identifying qualitative indicators
Some tips that may help you identify qualitative indicators:
It is easier to assess behaviour than feelings, as behaviour can be observed.
Ask the question, ‘If the project were a complete failure, how would people know?’ It is sometimes
easier to identify indicators for failure. The same indicators, looked at from a positive perspective,
can often be used as indicators of success."VWomen do not participate in public meetings’ might be
an indicator of failure. Looked at positively, an indicator of success could be,‘More women partici-
pate in public meetings’.
Have a discussion and ask, ‘What do you mean by... (participation, for example)?’ A discussion or
brainstorming session can identify how different people understand a concept. By breaking a con-
cept into more specific components, it becomes easier to select useful indicators. ‘Increasing the
participation of community members in the project’ is a concept that may not be well understood,
and for which there may be many different interpretations. By breaking the concept into more spe-
cific elements, it can become easier to identify how different project stakeholders understand the
concept and its application in the context of the target group and beneficiaries.
(Gosling 1995)
What do the indicators assess?
Each objective is associated with an indicator that helps in assessing different aspects of the
project’s progress. While objectives in the project plan describe outputs and outcomes, indicators
tell you what progress you are making in achieving your outputs and outcomes.
22
Monitoring and evaluation: tools for biodiversity conservation and development projects
Outcomes
Outcomes indicators assess progress towards the Project Purpose and Development Goal. They
are concerned with whether or not the project is making a difference. There are two questions
that indicators at an outcomes level are particularly interested in:
« Target group response: Is the target group demonstrating change?
» Are they aware of the project's services and products (its outputs)?
» Are they using them in the way intended by the plan?
* Benefits:Are beneficiaries receiving any benefits as a result of the project?
» Is the project making a difference for beneficiaries?
» How widely distributed are these benefits?
The project targets conservation managers working on sites in the City of Cape Town’s biodiversity network, by
providing them with various forms of support.
For its outcome indicators, the project wants to find out:
= Target group response—are the conservation managers who were exposed to the project’s support making
any changes in their conservation practice?
* Benefits—are there any benefits for people living on the Cape Flats? Can they be attributed to changes in
the practices of conservation managers?
Development goal Indicators
Parts of the CFR are conserved in a way that = Increase in priority areas under conservation
benefits people of the Cape Flats and is embraced management.
as a valuable element of urban life . nh # : :
f f = Increase in baseline jobs associated with con-
servation and nature-based tourism.
Project purpose Indicators
Sustainable conservation management of sites is = — Skilled people-centred urban conservation
achieved through active partnerships managers are running the sites.
* Local leadership drives community conserva-
tion efforts.
(Urban Sustainable Conservation Management Project)
Outputs
The outputs indicators assess whether the planned support to the target group is actually taking
place, and how this is being carried out. Its concern with progress is focused on two areas:
= Performance: How are the project outputs being achieved in relation to the plans? Has the
project met its targeted goals for:
Monitoring and evaluation: tools for biodiversity conservation and development projects 23
» Quantity—were the specified numbers of services or products delivered?
» Quality—were the outputs delivered to the required standards?
» Timing—were there any delays in providing the outputs?
» Location—were the outputs delivered in the targeted locations?
» People—did the right groups of people receive the outputs?
= Efficiency: Has the project made good use of its human and financial resources?
» Were too many resources used?
» Were too few resources allocated to activities? Did this affect the performance?
The project is interested in output indicators that will help it learn about:
= Performance—what progress has the project made in providing support to the Conservation Managers?
~
= Efficiency—is the project within the budgeted costs for the support it is providing?
Outputs
Sustainable conservation management is
demonstrated at four pilot sites
A campaign is conducted to gain broad support for
the four pilot sites
(Urban Sustainable Conservation Management Project)
e Enabling indicators
Indicators
Conservation management plans are devel-
oped in consultation with the surrounding
communities.
Certain activities (alien clearing, flora and bird
monitoring, and fire prevention) are carried out
in partnership with local communities.
Four community-based organisations with a
common values base, are introduced to the
project.
The profile of the four pilot sites is raised
through regular coverage in mainstream and
community media.
A pamphlet and video are distributed for each
site.
The enabling indicators assess the current status of the project's Assumptions. These are the ex-
ternal factors that are necessary for the success of the project but that are outside the control
of the project. In reviewing these indicators, two questions have to be asked:
» Are the external factors that were previously absent, now in place?
» Are the important external factors that were present at an earlier stage, still in place?
S
24 Monitoring and evaluation: tools for biodiversity conservation and development projects
The project's enabling indicators help it to be alert to its external environment. Does the external environment
support the services that the project provides to the conservation managers? Does the external environment
support the conservation managers in carrying out more sustainable conservation management practices?
Assumptions Indicators.
Funds are raised to support continued employment ® Signed contracts and agreements are in place
of the urban conservation management team with a number of donors.
There is reasonably low turnover in community = Most of the community leaders in place at the
leadership (allowing for capacity to be built) start of the project are involved in the sites
three years down the track.
(Urban Sustainable Conservation Management Project)
How do you develop indicators?
There is no generally agreed method for developing indicators. The previous pages have introduced
a number of issues to which you have to pay attention as you develop indicators for your project.
One way of developing indicators involves a three-step process:
|. Make sure the objectives specify what you plan to achieve
Draft objective: Sustainable conservation management.
Specified objective: Sustainable conservation management is demonstrated at four pilot sites within the City of
Cape Town’s biodiversity conservation network.
(Urban Sustainable Conservation Management Project)
2. Identify the components of the objective that you want to measure
More than one indicator will sometimes be needed for each objective. You may need indirect or
proxy indicators in some cases.
Objective Indicators
Sustainable conservation management is = User-friendly annual conservation manage-
demonstrated at four pilot sites within the City of ment plans are developed and implemented at
Cape Town’s biodiversity conservation network two sites.
# Infrastructure needs are identified and funded
at two pilot sites.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 25
3. Establish how you will collect this evidence
Some of the measures you have identified as your indicators may be very difficult to collect, and
you may struggle to gather this information. If this happens, you will not be able to verify (prove)
whether the planned changes are taking place, and you should return to the previous step and
look for different indicators. If you think you will be able to collect it, then your access to this
source of information will become your means to find out what is happening. You can use it to
check or verify whether your objectives and intermediate targets have been accomplished.
\))
Objective Indicators Means of (GF
Sustainable conservation = User-friendly annual conserva- verification
pignazement a demonstrated tion management plans are = Annual conservation man-
a four pilot sites oe the developed and implemented at agement plans.
City of Cape Town’s biodiversity AVelsies:
conservation network = Area manager reports.
" Infrastructure needs are identi-
fied and funded at two pilot FOR ERS Ts
sites.
(Urban Sustainable Conservation Management Project)
In Section 3 we look in more detail at methods of collecting the information.
Some common challenges with developing indicators
e Developing too many indicators
The purpose of indicators is to help you assess the status of your objectives. You need just
enough indicators to provide you with this information.
The risk in developing too many indicators is that:
* It will be time-consuming to collect all the information you require.
= Some of the information you collect may be irrelevant for the purpose of supporting appro-
priate and timely decision-making about the project.
e Impossible measurements
Some indicators may be beyond your capacity to measure. This may happen if:
« The objectives are not clear.
" The information will be too difficult or expensive to produce, or is not available to you from
other sources.
= The information cannot be collected within the lifetime of the project.An example of this
is whether or not beneficiaries continue to benefit from the project's outcomes over an
extended period (sustainability).
26 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Measuring the wrong levels
Each indicator should relate to a specific objective. If you try to use the same indicator to
measure more than one level of the project, you are unlikely to get information that will help
you to manage the project.
Gender-sensitive indicators
By now, it is well known that women and men can experience the same situation in different and spe-
cific ways. Similarly, your project intervention might target men and women differently.
There are many factors that can be examined to help explain why development impacts differently on
women and on men. One factor that is relevant in the context of M&E, is that few projects use gender-
sensitive indicators (ones that distinguish between women and men).
The value of developing gender-sensitive indicators can include:
* Including women’s contributions in assessment of the project’s outputs.
whether women are benefiting from the project.
= Assessing whether or not women are carrying an additional burden for household and community
work.
If your M&E plan is to include gender-sensitive indicators, it will be important to distinguish between
the gender of respondents when gathering information. This is an important step towards ensuring that
women are not disadvantaged by development projects.
(CIDA 1997)
r
!
'
1
!
!
!
'
i
1
!
1
!
1
1
1
'
'
1
1
'
1
'
1 ® Identifying the adoption of the project results by women as well as men allows you to assess
i
i
:
!
f
i
!
i
i
!
!
1
1
1
!
'
t
i
i
i
‘
i
I
&
The selection of indicators can be a political process. Indicators are subjective, both in terms of
how they are selected and how they are interpreted. The use of quantitative, scientific tools does
not necessarily make the information more objective and beyond dispute. Laboratory tests to
assess water quality may be biased by the localities from which the water was collected, the time
of the day or week when it was collected, and by the kind of tests carried out.
It is sometimes necessary to have more than one indicator for each objective. Each indicator
can measure a different aspect of the objective and, together, they can make a more convincing
argument about the status of the objective. Indicators are a tool to help you measure changes and
to communicate them to people. It would be better to select more appropriate indicators (which
stakeholders agree will measure the changes important for managing the project) than to argue
whether the indicators are valid ways to assess the objectives.
Monitoring and evaluation: tools for biodiversity conservation and development projects 27
Exercise: Reviewing output indicators
Think of one of the project outputs on which you would like to show progress. Consider the following
questions in relation to the indicators for that output:
Do they contain elements of quantity, quality, time, location and people (QQTLP)?
Do they tell you enough about what you have to measure?
Do you have any new information that you can use to update these indicators and will make their
formulation clearer?
Do you have a method for collecting the quantitative data?
Do you have a method for collecting the qualitative data?
In the light of these questions, would it be useful to re-formulate these indicators?
28 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.4 WHAT METHODS WILL BE USED TO GATHER INFORMATION?
To have timely, relevant and trustworthy information that you can use to assess whether the de-
sired changes are taking place, you will have to use appropriate methods for information-gathering.
If the methods are not appropriate or feasible (including cost, time and expertise), you will not be
able to measure the objective. Without any feedback about what is taking place, it will become very
difficult to engage in informed decision-making, accountability and learning.
Using legal milestones and technology
‘Our M&E has three aspects to it. Firstly, progress on the number of stewardship sites is monitored by legal
tracking, of where contracts are in terms of signing, et cetera. Secondly, the increase in the extent of land covered
by stewardship agreements is monitored on a GIS. (These metadata include quite a lot of detail about each site.)
Finally, there’s the information that is needed for reporting to C.A.P.E. which is captured on a spreadsheet.’
(Conservation Partnerships Programme)
How can you select methods?
Selecting a suitable method depends on a number of factors. Questions to help you select
appropriate methods can include:
e What information is required?
What questions do you want answered? How will the information be used? For problem-solving
purposes, understanding why a problem exists can often be more helpful than learning how
widespread it is.
e What context and medium of communication will be appropriate?
Some of your information needs can be met by individual informants, while other needs can be
satisfied by groups. This individual-group context should inform the methods that will be most
appropriate. Additional factors to think about include whether or not the information will be
gathered from stakeholders through:
* Written responses (questionnaires).
* Oral means (interviews and surveys).
* Visual means (maps).
* Drama (role plays).
* Story telling.
e What unit of analysis will be used for information-gathering?
The scope and scale of the information-gathering work will also influence the methods you
will use. For example, you will need different methods if you are gathering information from
individual households in a community compared to gathering information from the community
as a whole.
Monitoring and evaluation: tools for biodiversity conservation and development projects
You may also have to use different methods if you are working across a number of geographic
areas.
e What resources are available?
While quantitative methods are often more costly in terms of equipment, people and time,
qualitative methods require specific skills. Qualitative methods are often more suited to
contexts in which time and resources are limited.
Section 3 contains further information on some of the issues and methods involved in gathering
data.
Triangulation—using more than one source of information
A challenge that must be faced by all people who collect information, is the question of reliability. How
do you know if you can rely on the information that your informants have given you, especially if it sug-
gests that you have to make changes to the project plan?
One way to be sure the information is reliable and to reduce bias (whether in the collectors or the
informants) is to use more than one method to collect the same information. This approach to cross-
checking qualitative information is called Triangulation.
There are different ways of doing it:
# Use a multidisciplinary team to collect the information.
‘The interviews were conducted by a plant ecologist and a social scientist.’
# Use different tools and techniques.
‘In addition to the interviews, a number of photographs were taken of the land that the farmers are
utilising, to see how this compared with what they said about the land in the interviews.’
® Collect information about the same issue from different sources.
‘Information gathered during interviews with departmental officials was verified at the group
interview with the farmers.’
Triangulation can be especially important when you are assessing the project’s outcomes. It helps you to
establish whether the target group is changing as you planned, and whether you have to make changes
to your work as a consequence.
(Gosling 1995)
29
30 Monitoring and evaluation: tools for biodiversity conservation and development projects
1.5 WHO WILL PARTICIPATE IN Mae?
Making participation meaningful
‘We have an active multi-stakeholder steering committee. There are 28 members who meet every quarter. It
has been really important to make this committee accessible to community members. To make this possible,
we started an Outreach Programme which has three components to it—firstly, we provide opportunities for
community representatives to really explore the corridor, to become familiar with the issues. Then we ask them
to sit on the committee and to present something from their experience, to bring their experience into the room.
Lastly, we set up “knowledge exchanges” where they attend workshops or conferences, and really get to learn
from other people. This has made active and meaningful participation possible. If we are not focused enough on
transformation issues, for example, Dominee Hans will ask us about this!’
(Biodiversity Corridors Project)
There are many examples of projects in which the M&E work has failed to ring warning bells about
looming problems. One reason why this happens is that the M&E plan does not reflect sufficient
understanding of the local context. This may become an issue when the plan has been developed
and run by outsiders (such as planners or donors) or by specialists (such as researchers and aca-
demics).
If the M&E plan is developed and carried out in a way that involves project staff and other local
stakeholders, it can bring about a number of benefits.
e Timely and relevant information can be collected from stakeholders
With an improved understanding of the role of M&E in supporting project implementation,
stakeholders may be more willing to provide appropriate information.
e The local context can better be taken into account
By paying attention to the social and political context in which the project is being implement-
ed, and the value systems in which it is located, information can be more clearly understood in
context.
e Any lessons and decisions will be owned and acted on locally
This can have positive consequences for successful project implementation, as staff and other
stakeholders understand the reasons behind any decisions to make changes to the project's
implementation strategy.
Which stakeholders should participate in monitoring?
Stakeholders will make contributions at different points throughout the project. You do not want
to over-burden stakeholders by involving them in gathering and assessing information that has lit-
tle direct relevance to them, as it might result in them becoming less willing to contribute as time
passes. However, including them in a meaningful way is key to building ownership of the project.
Monitoring and evaluation: tools for biodiversity conservation and development projects 3|
The nature of the relationship between the development practitioner and the stakeholders will
determine the quality of exchange—and therefore the quality of information.
You can identify the aspects of the project in which the participation of key stakeholders will be
important. You can then assess:
= The relevance for each group of being involved in monitoring; what they can contribute and
how it will benefit them.
=» Whether or not they will be end-users of any resulting information; some end-users may want
to be involved in gathering the information used to propose changes.
= Whether or not any specialised technical capabilities will be needed. Be clear on the role stake-
holders will play in relation to each other in the M&E process so as to maximise participation
and not exclude those participants who do not have these technical capabilities.
32 Monitoring and evaluation: tools for biodiversity conservation and development projects
4 Ay Exercise: Mapping the participation process
Think about the different phases in the M&E process. Identify which stakeholders should partici-
pate at the different stages of your project:
|. Develop the M&E plan
. Gather the information:
» Outcomes.
» Outputs.
» Enabling factors.
. Analyse the information
4. Act on the analysis:
» Learning.
» Decision-making.
» Accountability.
(Abbot & Guijt 1998)
Monitoring and evaluation: tools for biodiversity conservation and development projects 33
1.6 WHEN WILL Ma TAKE PLACE?
Although M&E is an ongoing process throughout the project, it is still necessary to schedule spe-
cific M&E activities. This can help to ensure that the information will be available when you need it
to inform a review of the work, and to support timely learning, decision-making and accountability.
As you think about when you will have to review various aspects of the project, there are a few
points to keep in mind:
= If you review progress too frequently, you will end up devoting large amounts of time and other
resources to the monitoring process. This can result in the DRIP syndrome (Data-Rich, Infor-
mation-Poor).
# If you do not review key indicators often enough, you may miss important information and
trends. lf this happens, you may not be able to prepare timely responses to changing conditions.
Many host programmes have a schedule for reporting at regular intervals. If you already have such
a system in place, you may be able to adapt it to meet the M&E needs of your project. However, it
would probably not be wise to create a new M&E system that you use alongside any existing re-
porting system. In addition to the extra resources it will require, it is also likely to create resistance
among staff who may feel that they spend more time gathering information and writing reports
than implementing the work of the project.
How do you establish the timing?
To establish when you will have to gather information about the various parts of the project, you
can explore the following questions:
e Do you need specific information for regular management meetings?
You may have a Project Steering Committee or a similar group that meets to review progress
and consider if any changes to the project plan are needed. You should plan to gather the need-
ed information far enough in advance of scheduled meetings so that you will have enough time
to analyse the information and prepare your recommendations. Care should be taken that it
is not done so far in advance of the meeting that the information is out of date and no longer
relevant by the time it is being considered by the decision-makers.
e Do you have to know when a particular activity is completed?
Some of the activities in your project or operational plan may be dependent upon one anoth-
er—certain activities cannot take place until earlier ones have been completed.
You can plan to assess if the activities have been finished on schedule, and if the quality of what
has been accomplished provides a sufficient foundation for the next step of the work.
34
Monitoring and evaluation: tools for biodiversity conservation and development projects
Do you have to reflect on progress at key review dates?
Donors and other stakeholders may require you to submit an annual report, outlining the
progress you have made in implementing the project and identifying the outcomes being
achieved. This is an important opportunity to reflect on the change that has happened over
time, and to gain support from the donor for any changes that might be needed in the plan.As
the work is still in progress, you can assess intermediate targets—reviewing whether or not
you have made the progress you expected at this point of the project.
When will informants be available?
Reviewing progress on the project outcomes will often require information from representa-
tives of the target group and the wider beneficiary population. There are a number of factors
that might influence whether or not people will have sufficient time to participate in M&E
activities. For example, there are annual financial year-end demands on some staff.
e Are project staff available to carry out M&E work?
The M&E work has to fit into the project schedule along with other activities. You may have to
adjust the M&E schedule to accommodate staff as they implement key project activities.
The M&E schedule is often driven by the need for accountability and producing reports. Ideally,
it should be driven by the need for reflection and learning, to inform the development approach
being implemented by the organisation’s development practitioners or the project team. Once
the M&E schedule has been developed, it should be managed like other aspects of the plan by
accommodating flexibility. The project will be assessed on whether it achieved its objectives, not
on whether it produced all its reports on time.
Use the M&E schedule as a guideline, while placing your focus on the areas where it is needed
at the time. It would be better to negotiate changes in report deadlines than to miss key imple-
mentation targets.
Monitoring and evaluation: tools for biodiversity conservation and development projects 35
1.7 HOW WILL THE M&£ SYSTEM BE MANAGED?
Managing proactively and creatively
‘I make sure that my projects deliver. This is very important. | am also open and up-front when things don’t go
as planned. It is really important to let donors, for example, know when something is not going as it should. But
sometimes it happens that you have allocated one month to hire a manager and it ends up taking three months
to find the right person. Or the person has to give a longer notice period. But it is always about credibility. About
maintaining the credibility of the project.And managing creatively so that you can come as close as possible to
delivering on that target. Sometimes this means re-allocating resources—people, time, money.’
(Biodiversity Corridors Project)
The project M&E system that you are developing should be appropriately managed. Along with
other components of the project and its management system, the M&E system will need resources
and management support to ensure that it is carried out effectively and in a way that contributes
to the overall success of the project.
There are a number of problems that can result from a poorly managed M&E system. These prob-
lems can harm the project and its reputation among stakeholders. They may include:
" Failing to collect information on time.
= M&E work of poor quality.
= Overspending the budget.
# Limited capabilities.
In addition to the above problems, appropriate management is necessary to ensure that the project
can respond to unexpected developments. The demands of a dynamic context may call for un-
scheduled assessments to inform the future direction of the project.
What has to be managed?
As part of the project management system, there are a number of elements that will require atten-
tion.
e People
= Training and supervision of staff for collection and analysis of information.
= Training of managers to promote the effective use of information for learning and decision-
making.
= Supervision to maintain standards of monitoring.
e Systems
= Integration of the information into the management structure.
= Meeting data storage, processing and retrieval needs.
= Clear formats and timing of reports.
36
Monitoring and evaluation: tools for biodiversity conservation and development projects
Use of the monitoring information
* Learning by staff and partners.
= Management responses to unexpected developments.
* Accountability to stakeholders by seeing to it that reports are disseminated timeously.
Time
* Sufficient staff time to conduct monitoring activities.
* An early investment of time to build staff capability and stakeholder participation, which is
reduced at a later stage.
Financial
= The cost of designing the system.
« The cost of staff time to collect and analyse information.
= Resources to print and distribute forms.
* Resources to modify and improve the system.
Monitoring and evaluation: tools for biodiversity conservation and development projects 37
1.8 HOW WILL LEARNING BE BUILT INTO THE Ma PROCESS?
Learning sits at the heart of good M&E practice. It is what transforms the information that has
been collected and analysed into new actions and improved practice.
The Action Learning Cycle provides one model for ‘learning while doing’. It focuses on our experi-
ences (or the experiences of others), and on reflecting and learning from them.
‘
g
R
The Action Learning Cycle
Experience
Planning Reflection
Insight
The elements of the cycle are:
Experience
This comprises the concrete actions and experience of the project. It could be the experience of
implementing the work, or it could be information about the context of the project.
= What has happened?
= Who was involved and what did they do?
Reflection
The process of looking back on the experience. This is often done by comparing the experience with
what was expected, and looking for similarities and differences.
= ~=Why did it happen? What caused it and what helped it?
= How does it differ from what | expected to happen?
38 Monitoring and evaluation: tools for biodiversity conservation and development projects
Insight and understanding
This occurs as we make sense of what has taken place and become aware of its significance for our
plans. This insight can make us wiser!
* What could have been done differently?
# What did | learn? What new insights do | have?
* What new questions do | now have?
Planning new action
!
'
U
'
'
‘
i
i
1
1
1
'
The insight leads us to consider appropriate responses. By acting on our insights, we can prevent 1
‘analysis paralysis’ from holding us back. 1
=# What do | want to do?
# What am | going to do differently? t
i
For learning to take place and to result in improvements in the way we do things, we have to go 1
through all four elements of the action learning cycle. Without insights, no new meaning and under- '
: : : 1
standing can result from the reflection, and we may continue to repeat mistakes or be blind to an exter- ,
nal factor influencing our work.We also have to pay attention to the step of planning, to ensure that we 1
identify concrete ways in which we will do things differently the next time.
'
1
'
u
i
i
é
1
a
As soon as the action learning cycle is completed, we commence a new cycle that reviews our latest
experiences. Rather than being isolated events, action learning provides us with an ongoing series of
linked cycles, or a spiral, in which we are learning and continually improving our approach to our work.
(Taylor et al.1997; Raelin 2000)
The Action Learning Model is one approach to learning. There are many others.
Ss Knowledge exchanges
«
| think ‘knowledge exchanges’ are a great way to learn. It gets rid of blinkers. It exposes people to what others
are doing, to new things. You can learn from others. The important thing is that you must bring it back and share
it. Sometimes you realise that others are not so far ahead of you.This can also be a good learning!
(Biodiversity Corridors Project)
For learning to be possible in a project or organisation, there are certain enabling conditions. One
model (Soal, 2007) identifies the following features as being important components of a learning
environment:
= Space—creating sufficient dedicated time for reflecting and learning together about your prac-
tice.
= Rhythm—regularly coming together to learn so that there is a rhythm in this, something which
is compelling and robust.
» Having a champion—someone who has the authority and will to make it happen. This is particu-
larly important in the early stages of setting up internal learning.
# Having an approach—a way of learning. This could be through case studies, through presenta-
tions, by doing research, having conferences etc.
= Collegiality—a sense that you are trying to build or contribute to a purpose that is bigger than
the sum of the individuals who make it up.
Section 2 explores learning further.
Monitoring and evaluation: tools for biodiversity conservation and development projects 39
Exercise: Application of the action learning cycle
Think about the following questions:
" How would you describe the action learning cycle to others?
= What are the benefits of this kind of approach to learning?
=# What is its relevance to M&E?
Make notes, and share your thoughts with a colleague.
In Summary,
In Section | we introduced a number of questions to guide you in developing your M&E plan.
They were to assist you to:
* Clarify the objectives for your project.
= Establish who the end-users of the M&E work will be.
= Develop indicators to help you assess success.
= Identify methods for information-gathering.
= Identify which stakeholders should participate in project M&E.
= Establish the timing of M&E activities.
= Clarify how the M&E system will be managed.
= Recognise the importance of learning in M&E.
The M&E plan guides you in carrying out project M&E activities and processes. It should be
flexible, and should be updated as your project plan changes, and as you learn improved ways of
working.
The M&E plan can also serve as a tool for communicating with project stakeholders—
externally (community representatives, donors and partners) and internally (management and
staff). This can inform the stakeholders of the roles that they might be expected to play, and
when they can expect to receive progress updates from the project.
REFERENCES AND FURTHER READING
ABBOTT, J. & GUIJT, I. 1998. Changing views on change: participatory approaches to monitoring the environment. SARL
Discussion Paper No. 2. International Institute for Environment and Development, London. http://www.iied.org/
pubs/pdfs/6140IIED.pdf (accessed January 2008).
CAVILL, S. & SOHAIL, M. 2007. Increasing strategic accountability: a framework for international NGOs. Development in
Practice 17,2.
CIDA 1997. Guide to gender-sensitive indicators. http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Policy/$file/
WID-GUID-E.pdf (accessed January 2008).
GOSLING, L. & EDWARDS, M. 1995. Toolkits: a practical guide to assessment, monitoring, review and evaluation.
Development Manual 5. Save the Children, London.
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ.
RANDEL, M. 2002. Planning for monitoring and evaluation. Olive Publications, Durban.
SOAL, S. 2007. Towards ‘better evaluation’-—an account of one internal practice. Keynote address to the Australian
Evaluation Society (AES) Conference, September 2007. Melbourne.
TAYLOR, J., MARAIS, D. & KAPLAN, A. 1997. Action learning for development: use your experience to improve your
effectiveness. Juta, Cape Town.
WWE (World Wide Fund for Nature) 2005. Basic guidance for Step 2.2: monitoring plan. http://assets.panda.org/
downloads/2_2_monitoring_plan_0|_1|1_05.pdf (accessed January 2008).
40 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 2: LEARNING
In the introduction, we described M&E as a process of ‘learning while doing’. This kind of learning
draws directly on our experience. Such an approach to M&E means that change and adaptation are
core values and that the insights gained from previous experience are built into future actions. This
links closely to the notion of ‘adaptive management’, which is a familiar management tool within the
biodiversity conservation sector.
It is this approach to learning that underpins this handbook.
In this section, we focus on the learning that happens in the project team as a way of improving
your biodiversity conservation practice and delivering outputs that have greater benefit. This lays
the foundation for future steps or phases in the M&E process.
As with other aspects of project M&E, learning can impact on your objectives at two levels:
* At the level of outcomes—the difference the project is making in the lives of the target group
and beneficiaries.
* At the level of outputs—the products and services the project is offering to the target group.
Within each of these areas of objectives, there are two questions to consider:
= Are we doing the right things?
= How do we do things right or better?
By the end of this section, you will have:
* Explored the importance of learning within the context of M&E and some of the ways in which it
takes place.
Identified ways of facilitating learning within the project team.
* Amore in-depth understanding of the case study method and how it could be applied in your con-
text.
a
1
f
1
1
1
1
1
1
t
i
1
1
# Ideas for building a learning environment.
1
|
Monitoring and evaluation: tools for biodiversity conservation and development projects 4\
2.1 HOW DOES LEARNING HAPPEN?
A useful learning concept is the notion of single- and double-loop learning. It draws our attention
to the importance of un-learning and re-learning, and some of the barriers we might experience to
new ways of doing things. These barriers are often experienced as challenges to our existing ‘men-
tal models’.
Challenging our mental models
What happens when we come from different places
‘One of the challenges for us was around entrenching an external project (which had been operating for
a while and had its own way of doing things) into an existing organisation (which also had its own culture
and way of doing things).We thought the business units were slow in understanding and incorporating the
project’s ideas and approach into their operations. They probably felt the same way about us! An “us and
them” mindset seemed to get in the way.
(Conservation Partnerships Programme)
This example illustrates the power of our mental models at work. Mental models, or paradigms, are the
ways in which we understand how the world works. Our repeated experience of seeing certain things
happen again and again, makes us form a mental model that tells us that such things will always happen
in this way.
The project staff had a mental model for how the project should be implemented. The staff at the busi-
ness units had a mental model for how they approached their work.At some sites, even though the
business unit staff were aware of the achievements that the project was showing elsewhere, they were
unable to overcome their ‘paradigm blockage’ because they were attached to their standard way of
thinking and could not integrate the new ideas into their work.
When we are analysing the M&E information, we are at risk of experiencing paradigm blockage.As we
look for the causes for the changes we are being made aware of, we may find ourselves limited by the
boundaries of our mental models.We may not even know we are being limited because it is very dif-
ficult to be aware of what forms our mental models.
We may be able to challenge our mental models if we consider how we look at the information that
lies before us.
e Single-loop learning
In this kind of learning, a response to an observation or event is based on a person's (or
organisation’s) existing set of values, beliefs, norms. If the activities we embark on do not produce
the results we expect, we change the activities in the hope that it will bring about the desired
results. This is an ‘error correction’ approach to learning.
e Double-loop learning
In double-loop learning, our responses emerge from a reflection on dominant or stated values,
beliefs and norms. The information that things have changed provides a challenge to the thinking
that lies behind our current activities.As the activities did not bring about the expected results, we
question our assumptions and the mental models that led us to these activities in the first place. We
re-think the rules, so to speak.
Fr
4
i
1
i
I
1
1
1
i
1
1
1
I
1
1
1
i
1
i
1
i
I
I
4
i
1
1
Q
i
i
:
i
1
4
j
fl
i
1
I
1
1
1
1
i
1
i
f
1
'
1
1
1
I
i
i
‘
1
I
I A aaaes : : :
1 The value of double-loop learning is that it helps us to step back from our habitual reactions to
\ have a critical look at our mental models. This may lead us to question the underlying assumptions
that inform the project’s plans.
i
42 Monitoring and evaluation: tools for biodiversity conservation and development projects
~~" sss se se es se eet erhlUCcrrhlUlCUcrTClUC rClClCUcrClCcrlClClCcrrlClCchrlhlClCcrlClCchOrlClCchrrlClCcrrlhlClCcrOrlhlClCcrrlhlClCchrrlhlCchrrlhlCcrOlhlCUcrhrlUcrrlClCcrlClCcrlClCcrlhlCchl rT OC rOllCUcrOlClCcrlClCcrlClCcrrlClCcrrlClUCrrlClCUcrhrlUcrlUcrlUcrlU |
Two other concepts that are similar to single- and double-loop learning are patterning and puzzling:
e Patterning
In this process of learning, we develop meaning by linking our current experience with similar ones
we have had in the past—this is learning by generalisation.
e Puzzling
Puzzling happens when we are confronted with an experience that we have not had before or that
contradicts previous experiences—this is learning by exception, contradiction or surprise.
(Raelin 2000; Smillie & Hailey 2001; Uphoff & Combs 2001; Rogers & Williams 2006)
There is a place for these different kinds of learning, including single-loop learning and patterning
which, for example, enable us to become adept in a particular skill. It is when we are stuck that
we may have to turn to other ways of learning. While double-loop learning is not necessarily more
time-consuming, it can be much more challenging than single-loop learning. It asks you to look criti-
cally at your basic beliefs and assumptions about the project’s goals and context as well as at your
current way of working. ‘Puzzling’ is similarly challenging.
Helping people to learn, with these concepts in mind, requires skilled facilitation. Managers must
strive to find a balance between support and challenge in helping people to let go of old ways of
thinking and doing, and taking on new ones.
In your role as a project manager, you may find it useful to bear this in mind when working with
people in learning situations.
Monitoring and evaluation: tools for biodiversity conservation and development projects 43
2.2 LEARNING IN THE PROJECT TEAM
Encouraging learning in the project team is a key issue. By helping project members to develop new
insights and understanding, you will be able to contribute to improved work practices and hence to
improved outputs that you can offer to the target group.
Learning can come from many different directions. Our day-to-day experience in the field is an
important source of learning. So is the wider context within which we implement this work.
Case studies are particularly valuable for looking at the experiences and practices of staff ona
daily basis. This is the tool on which we focus here. Some of the other ways in which the team can
learn are described briefly. They are: after action reviews, peer assists, debriefing, team meetings and
workshops, and benchmarking.
Exercise: Trying a new tool
. Can you think of a recent question or problem within your work environment with which you have
been grappling that would benefit from being presented as a case study?
. Spend 20 minutes writing it up, taking into account the following questions:
What happened?
Who was there?
What did you do?
What were you thinking and feeling at the time?
What were the results of what you did?
What new questions and thoughts do you have?
. Identify three or four colleagues who have a similar commitment to learning as you do, and tell them
what you know about case studies (see below).
. Ask them if they would be willing to try out this new tool with you.
. Agree different roles for the first session.
. Decide on a time and place.
. Debrief at the end of the first session, and test if there is any interest in trying it again.
Using case studies to learn about your conservation practice
e What are case studies?
Case studies are particularly useful for gathering information on your current work practices,
for describing what you are doing in the field and for learning about what works and does not
work. Using a case study is an opportunity to present a persistent question or problem and to
explore and solve this problem with others.
44
Monitoring and evaluation: tools for biodiversity conservation and development projects
Case studies, described in this way, contribute to the monitoring of performance-related as-
pects of a project’s outputs. They help to answer the questions “What progress is being made in
providing goods and services to the target group?’ and ‘What are the standards to which they
are being delivered?’ Case studies can also reveal any delays there might be in delivering these
services. They can deal with any aspect of quantity, quality, timing, location and people (see Sec-
tion |.3).
There are four components to a case study
* Getting into the experience—reminding yourself of a particular situation, what happened,
what you did, thought or felt, what the results were and what new thoughts and questions
arise.
* Describing this experience by writing it up as a case study.
« Sharing the case study with colleagues by presenting it.
* Having a discussion in which colleagues listen, ask questions and provide input, generating
new perspectives and angles on the issue in question.
The skill of listening
Central to all of these components, is the skill of listening. It requires that, as a colleague, you
are able to hear what the presenter is saying without judging it. This kind of listening is demand-
ing—it is listening with the purpose of understanding the whole story and what the presenter
is telling you. There are three levels at which listening has to happen in order to hear the whole
person:
= At the level of ideas, thoughts and facts.
= At the level of feelings, reactions and emotions.
# At the level of will and intention.
Most of us are used to giving advice, to being an expert, to having the answers. When we work
with our colleagues in a listening mode, it is less about us and more about them.
In running a case study session, there are three key roles:
Chair and time keeper. Purpose is to:
# Decide how long you have for each case study (including presentation and discussion) and
to keep to the time allocation.
= Ensure that the presenter of the case study is not interrupted.
= Encourage participation and give space for discussion, being careful that one person does
not dominate.
* Try and follow the threads of the discussion.
= Summarise the discussion and a possible way forward.
Case study presenter. Purpose is to:
= Read the case study and the questions, slowly and clearly.
= Bring a level of distance to the discussion and to be open to the outcomes of the discussion.
In doing so, he/she does not have to respond to every point and to defend his/her choices.
He or she should allow the discussion to flow.
* Really listen to what his/her colleagues are saying.
Monitoring and evaluation: tools for biodiversity conservation and development projects 45
Note-taker and reminder of why we have case studies. Purpose is to:
= Remind everyone at the outset to listen at the different levels, and to work at really hearing
and understanding the presenter’s story.
= Remind people not to judge language or style.
* Take notes of the discussion.
These notes are important in generating information for more conventional forms of learning
and sharing, such as report writing to funders, for annual reports et cetera.
The benefits of using case studies
As a result of this process, you will have generated valuable lessons on what is and is not work-
ing for both the individual practitioner and the project as a whole. This can lead to important
improvements in the delivery of goods and services to the target group.
After Action Reviews
This is a way for a team to reflect on a piece of the work that has recently been completed, and
to draw out lessons. It can be used by the whole team as well as by subteams and individuals.
(This is elaborated in the box below.)
Peer Assists
A team starting a new project or preparing for an event in which they have limited experience
can call on people with experience for help. They can be invited to join the team in a meeting
or a workshop in which the guests tell their stories and share their insights and hypotheses.
The team’s aim is to learn from their peers about appropriate ways to approach the tasks lying
ahead.
Debriefing
This is a structured way for an individual or a team to reflect on their experience and to share
their insights with the rest of the team. It is best to do this at the end of an event or a phase
of work, as it allows the whole process to be reviewed. Debriefings can easily be recorded or
written up, allowing them to be shared with a wider audience at a later stage.
Team meetings and workshops
Any time that the team gather for a meeting or a workshop, there is an opportunity for learn-
ing. Part of the meeting can be used for reflecting on past events, making meaning of these
experiences, and drawing out insights and hypotheses for future action.
Benchmarking
This is a technique for learning from the experience and ‘good practices’ of other organisations.
It can be especially appropriate when you are moving into a new area of work or are trying to
improve the standard of your work. By looking at how other organisations do things, you can
identify the ‘benchmark’ or standard for the way you will do things.
46 Monitoring and evaluation: tools for biodiversity conservation and development projects
After Action Reviews
The After Action Review (AAR) is a tool for team-based learning.A team can use this tool on a regular
basis to develop insights into their experience and to make meaning of the information they are
gathering through their monitoring. The benefit of the AAR is that it does not require a large investment
of time or other resources.
The process for an AAR involves reflecting on a meeting, workshop or any project activity that has just
finished. Members of a team ask themselves:
=" What was supposed to happen?
* What actually happened?
=" Why was there a difference?
# What will we do differently next time?
The AAR allows the team to engage in ongoing reflection and continuous improvement without waiting
for formal monitoring to take place. Lessons can be identified and acted upon by the team, or passed to
others in the project or organisation where appropriate.
The AAR is an abbreviated form of the project monitoring process. It poses questions similar to the
ones asked during project monitoring.
By asking ‘What can we do differently next time?’ you can identify situations in which the lessons you
have just learnt can be relevant.
e Incremental improvements
If you are running a series of related activities, you can improve the way you approach later activities
by including any lessons from the earlier events.An example would be if you were running a series
of workshops for a group of people.An AAR will help you to identify ways to make each workshop
better than the previous one.
e Other related opportunities
The AAR lessons can be applied to situations that are not the same as the one you have just
finished, but for which the lessons are relevant. Lessons from one workshop can help you to
improve an unrelated workshop you are running for a different audience, or it may help you to
identify ways to publicise your work better.
(Collison & Parcell 2004)
G When learning becomes a habit
After each of our events or activities, such as Water Week or Dunes’ Day, we evaluate what worked or did not
work. We write this up and when we meet again, to plan another activity, we bring these notes and we use them
to plan better for the next event.We go back to these notes when we do our future planning. We are now in the
habit of doing these reviews cnd they really work.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 47
improving development and conservation practice
As mentioned in the introduction, one of the key M&E questions is, ‘How do we do things better?’
This goes to the heart of our development and conservation practice. One way of improving our
practice is to identify the level at which we are currently performing (the ‘condition’), and then
seek ways to improve this by identifying the ‘pathway’ that is appropriate for that condition.
Pathways to doing things better
1. You do not know if performance is good or
bad.
2. Performance is poor but you do not know
how to improve it.
3. Performance is poor but you lack the capacity
to improve it (resources or authority).
4. You know performance is poor but there is a
lack of incentives for improving it.
5. You are doing well but you do not know how
to maintain or replicate it.
(Rogers 2007)
Seek feedback or information immediately
and more regularly.
Seek expert knowledge or advice about
better ways to do things (knowledge
building).
Seek general support through advocacy.
Use carrots, sermons or visioning (use
incentives).
Learn from your success and how it can be
replicated, particularly through the process of
documentation.
To integrate learning into your work, the process of making changes requires the support of the
various stakeholders. This can help to increase the likelihood that the target group and donors, for
example, will provide their backing for the new measures, and that any necessary resources will be
made available.
It is helpful to keep in mind that the lessons you are formulating represent insights and hypotheses.
They are your interpretation of what has taken place, of how you have made sense of your experi-
ence. It is therefore useful to approach any planned changes as experiments. By treating them in
this way, you can pay close attention to what happens. If necessary, you can revise your hypotheses
over time, adding new information to confirm them or change them as needed. This will deepen
your adaptive management approach to M&E.
48
Monitoring and evaluation: tools for biodiversity conservation and development projects
introducing changes in your practice
The process of introducing change may occur in a variety of ways (Fowler 2000):
Continuous improvement and tinkering
The project makes minor changes as a response to learning from its experience. It can become
a key aspect of the project team’s culture over time.A risk you should be alert to, is that tinker-
ing with small changes may blind you to the need to question some of the central assumptions
held by the project (see the earlier subsection: Challenging our mental models).
When the project first started providing support to the conservators, they met with them often and were very
active in the work of the conservators.
This met with some resistance from the conservators who felt that the project was intervening too much.We
cut back on the frequency of the meetings.
(Urban Sustainable Conservation Management Project)
Reflective evolution
Changes can arise as a consequence of reflection becoming central to the project team’s way
of working. The stimulus for change is the reflection, not a crisis or the periodic monitoring
reviews. Reflection encourages a more formal recognition of what staff may already be feeling
intuitively. lt promotes proactive changes and renewal of the project strategy.
Over time, as the conservators became confident in giving feedback and the project’s management were able
to listen to it, the meetings with the conservators became more focused and purposeful. The project trained
the conservators in the practice of developing and presenting case studies. The conservators have found this
type of meeting to be very helpful. It provides them with support in the field and it creates a safe place for
them to share and discuss their challenges and problems.
(Urban Sustainable Conservation Management Project)
Process reform
The project may make changes to the way it goes about its work, without changing the role it
is seeking to play. Changes to the process may often occur in response to major changes in the
external environment. The project might stop providing certain services if it duplicates what
another organisation is doing. The project can then use its staff and resources to focus on other,
still neglected services.
While the project still aims to contribute to people-centred conservation management, local government has
now become better at supporting the conservationists that it employs. The project has stopped facilitating and
running the meetings with the conservationists. Instead, these are driven by the City.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 49
There are further possible degrees of change, such as role changes, restructuring and
transformation. These changes are more likely to involve far-reaching shifts in approach,
structure and staffing, and may not be relevant within the context of projects. These options,
however, might be relevant for organisations that manage projects as part of their work.
50
Monitoring and evaluation: tools for biodiversity conservation and development projects
2.3 BUILDING A LEARNING ENVIRONMENT
The context in which learning happens is integral to its effectiveness. In addition to the factors that
were identified in Section |, there are a number of other considerations in building a learning envi-
ronment. Roger Harrison (cited in Thaw 2002) identifies six factors that encourage learning.
There is constant attention to the learning process
The project or organisation values and encourages learning. People within the project team see
it as valuable.
There is an interest in finding longer-term solutions
Learning is not only aimed at solving immediate problems but attempts are made to understand
the bigger picture, as well as to distinguish symptoms from deeper causes.
Learning is self-initiated and self-directed
Staff are encouraged to take responsibility for their own learning and do not rely on others to
drive and identify learning opportunities.
People are able to think systemically
Because the situations in which people are working are complex, they must be able to focus on
the complexity of relationships, and on the whole, rather than on the parts.
Questions, questions and more questions
Learning works with questions, and questioning certain assumptions, values and practices of
the project or organisation. It does not ignore certain patterns—nothing is ‘un-discussable’ and
there are few ‘holy cows’!
There is a system in place to support the learning
This system enables the project or organisation to take in complex new information and
modify itself. It also has policies and processes in place to support individual learning efforts.
There are two common barriers to learning that project managers should keep in mind:
The presence of fear, anxiety and other strong negative emotions in the project team or organi-
sation.
The bias-for-action, which is a characteristic of most leaders and managers, and is prevalent in
the culture of the development sector.
Monitoring and evaluation: tools for biodiversity conservation and development projects 51
Exercise: Learning in your project
ca
Taking into account the factors identified above:
= Where do you see learning happening in the project? What and who are making it possible for
learning to happen?
Where is learning not happening? What and who are preventing learning from happening?
Why do you think this is so?
What do you think you can do to support and encourage learning within the project? How could
you influence it?
Do you have to take any steps to deal with negative emotions in the project (such as fear and anxi-
ety)? What steps could you take?
Do you have to take any steps to deal with a ‘bias-for-action’ within the project? What steps could
you take?
In| Summary,
In Section 2 we:
= Explored some of the concepts underpinning learning.
" Focused on learning within the project team by:
» Identifying the different sources of learning.
» Discussing some of the methods and tools that encourage learning, in particular active
reflection on own and others’ experiences.
» Identifying ways in which to introduce changes in professional practice.
" Looked at the use of case studies as a tool for capturing learning.
" Identified components of a learning environment.
REFERENCES AND FURTHER READING
COLLISON, C. & PARCELL, G. 2004. Learning to fly: practical lessons from one of the world’s leading knowledge companies.
Capstone, Oxford.
FOWLER, A. 2000. The virtuous spiral: a guide to sustainability for NGOs in international development. Earthscan, London.
HARRISON, R. 1995. The collected papers of Roger Harrison. McGraw-Hill, New York.
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ.
ROGERS, P. 2007. Building evaluation capacity and capability in organisations. Open day held at CDRA Offices, Cape
Town, 23 November 2007.
ROGERS, PJ. & WILLIAMS, B. 2006. Evaluation for practice improvement and organisational learning. In |. Shaw, J. Greene
& M. Mark (eds), Handbook of evaluation. Sage Publications, London.
SMILLIE, |. & HAILEY, J. 2001. Managing for change: leadership, strategy and management in Asian NGOs. Earthscan, London.
THAW, D. 1996-2002. Ideas for a Change Series. Olive Publications, Durban.
UPHOFF, N. & COMBS, J. 2001. Some things can’t be true but are: rice, rickets and what else? http://www.eldis.org/
fulltext/TRUE-RRR.pdf (accessed January 2008).
52 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 3: GATHERING THE INFORMATION
Information is the fuel that drives the project M&E system. It makes it possible to know what has
happened and what challenges may be looming on the horizon.A well-designed M&E system will
enable you to collect appropriate information on the work and.on the project’s context in a way
that will feed into timely and relevant decision-making, reporting and learning.
There are three broad considerations in the information-gathering process:
* What is the nature of the information needed for M&E purposes?
«= What are appropriate methods for gathering the information?
= How can you ensure that the information is accessible and useful for evaluation purposes?
A useful starting place is to refer back to the project plan. These plans provide a description of
what should be monitored (i.e. the indicators) and from where that information should be sourced
(i.e. the means of verification). They also remind you of what you said you were going to achieve.
The frequency of measurement may also be stipulated in the plan.
A further consideration is deciding who will be involved in providing and collecting information.
This has to be co-ordinated and managed.
By the end of this section, you will have:
« A better understanding of the kind of information needed to measure objectives at the level of
outcomes and outputs.
= Compared different methods for collecting data, and started to select methods appropriate to your
context.
" Been provided with some ideas on ways to manage collective information so that it is useful and
accessible.
Often when we are monitoring, we are counting the ‘blomme in the veld’. This is about numbers, about
what is quantifiable. We look at the number of hectares under conservation, how genetically different the
populations are. This provides us with rich detail. It helps us to be succinct in our communication.And while
this is necessary work, it is not sufficient. Sometimes we have to look at the relationships, to see what is
going on there, why things are not happening. This is much harder to measure.We use case studies to bring
these issues to the fore.
Mostly the information is in our heads or in files. We do not have to go and gather extra information. But the
process of extracting it can be laborious. It takes time.And making phone calls.
(Urban Sustainable Conservation Management Project)
Monitoring and evaluation: tools for biodiversity conservation and development projects 53
3.1 UNDERSTANDING YOUR INFORMATION NEEDS
Your search for information will be driven by the indicators you have developed (indicators are dis-
cussed in greater depth in Section |) in your project plan. They will direct you to the information
you will need to collect in order to assess what progress has been made for each of the objectives
you are monitoring.
e Information relating to outcomes
Information for monitoring outcomes tells you whether the project is having any positive influ-
ence on the target group and beneficiaries. It is information that is used for comparative pur-
poses. It usually compares the state or status of people or things over time and, sometimes, also
between different groups or study subjects at the same time.
Development goal Indicators «
Parts of the CFR are conserved in a way = Increase in priority areas under conservation
that benefits people of the Cape Flats and is management.
embraced as a valuable element of urban life g het 25 : :
Increase in baseline jobs associated with con-
servation and nature-based tourism.
(Urban Sustainable Conservation Management Project)
As indicators are typically used to measure the extent of changes that have taken place, it is often nec-
essary to know what situation existed at the start of the project. This starting point is called the base-
line and provides a point of reference against which to measure the changes. Baseline data can be col-
lected by quantitative or qualitative methods, depending on the kind of information you want to gather.
One problem that many projects face is that they do not have a baseline reference point, and so they
are unable to assess the extent of any changes that are taking place. Baseline information should ideally
be collected during the project planning process, and should form part of the rationale for establish-
ing the project. It is possible, however, to overcome the absence of such information by carrying out a
baseline study as one of the early activities of the project.
It can be complex to gather baseline information, and it can be a real challenge to know how typical
the measurements of the usual state of affairs are. In some projects, it may be more useful not to waste
resources on a baseline study. Instead, changes can be identified by comparing the first assessment with
later assessments.
(Feuerstein | 986)
e Information relating to outputs
Outputs are the results of activities undertaken during the course of the project. Output infor-
mation generally relates to the delivery of certain goods, services and products, and provides
answers to the questions, ‘Is the project making progress towards its objectives?’, "What is the
progress?’, and ‘How efficiently is the project using its resources?’ The information tells you not
only whether progress is being made but also about the quality and scale of the achievement.
54 Monitoring and evaluation: tools for biodiversity conservation and development projects
I outputs Indicators
A campaign is conducted to gain broad support " Four community-based organisations, with a
for the four pilot sites common values base, are introduced to the
project.
= The profile of the four pilot sites is raised
through monthly coverage in mainstream and
community media.
= A pamphlet and video are distributed for each
site within a year.
(Urban Sustainable Conservation Management Project)
As the outputs are dependent upon the internal activities of the project staff, much of the
monitoring information will be available through internal project documentation and other sec-
ondary material sources.An ongoing challenge for all projects is to verify that what is described
in the documents is what has actually happened. For example, staff may be tempted to inflate
the number of people who attended community meetings in an effort to please management
and donors by showing that there is strong community support for the project.
e Information relating to assumptions
In gathering information about the assumptions, you are looking for the presence or absence of
certain conditions. These conditions are outside the control of the project, but are critical to its
success. Information about the assumptions acts as an ‘early warning system’, drawing attention
to potential risks to the success of the project.Assumptions are frequently overlooked in an
M&E plan.
Fs
C Assumptions Indicators
Funds are raised to support continued = Signed contracts and agreements with a
employment of the urban conservation number of donors are in place.
management team.
There is reasonably low turnover in community = Most of the community leaders in place at
leadership (allowing for capacity to be built). the start of the project, are involved in the
sites three years down the track.
(Urban Sustainable Conservation Management Project)
As discussed in Section |, the selection of an appropriate method also depends on whether the
information that you need is qualitative or quantitative.
Monitoring and evaluation: tools for biodiversity conservation and development projects 55
e Qualitative information
This is information that tells you about human or social behaviour and perceptions. Qualitative
research generally provides answers to ‘How?’, ‘What?’ or ‘Why?’ questions and is recorded in
narrative format, for example as stories or case studies.
e Quantitative information
Quantitative information generally provides answers to questions of ‘How many?’, ‘When?’ and
‘Where?’. This is the type of data scientists are most familiar with. lt may be presented in the
form of statistics or graphs. Often it is percentages or other relational measurements, as op-
posed to numbers, that hold significance as indicators.
‘In this project, we have to maintain a certain number of kilometres of road.’
This is quantitative information that can be collected by measuring the length of the road on which
work has been done.
‘What we have also realised is that the roads have to be maintained to a certain standard. One expert came
and told us that when the rains come, these roads will be washed away.’
There are certain technical standards for road maintenance that can be used to establish the qualitative
indicators for this output. They tell us the ‘how’ aspect of road maintenance.
(Project Developers’ Forum, |8—19 March 2008)
The type of M&E that we are discussing in this handbook, what we referred to in the
introduction as ‘effectiveness measurement, is likely to require a combination of both
qualitative and quantitative information.
You should also find out if you will have to collect the information yourself (primary
information) or if the information is already available in a format that you can use (secondary
information).
e Primary information
This is information that you will be the first to collect. Selecting appropriate methods, you will
gather this new information directly from project participants/stakeholders, and from the target
group and project beneficiaries. You may even gather information from the general population
in which the target group is located, especially for information to help you assess outcomes
indicators.
e Secondary information
This is information that already exists, collected by other people or other organisations. Unless
it is material existing within your own organisation, it is likely that you will have access to this
information only in its completed form—as reports and articles, rather than as survey respons-
es and interview notes.
56 Monitoring and evaluation: tools for biodiversity conservation and development projects
Generally, gathering information from secondary material is likely to be less costly than undertak-
ing primary research. In some cases, secondary information will be sufficient to meet your needs.
In other cases, there may not be any secondary material available, or it may not be sufficient or
suitable for your information needs. You will have to select an appropriate method to conduct the
research that will provide you with the required primary information.
In considering the options, remember that you are looking for indicators, and indicators are just
that—they provide an indication of the state of things. They are not intended to be exact measure-
ments. They provide an indication of a shift or change. It may be impossible, for example, to do a
head count of every bus load of visitors, but a record of the number of buses could be maintained
and an estimate of the number of visitors extrapolated from that.As a guiding principle, some
information is always better than none, no matter how unscientific it may seem, as long as it is
consistent.
Monitoring and evaluation: tools for biodiversity conservation and development projects 57
3.2 SELECTING AND USING APPROPRIATE METHODS
Depending on the nature of the indicator, a wide variety of methods are available for information
gathering purposes. For example, interviews are useful for gathering information on people’s per-
ceptions and views on a particular issue, whereas a camera may help to capture hard evidence on
the physical condition of something. The challenge is to identify relevant and appropriate methods
that will help you get the information you need to analyse progress.Appropriate methods can be
selected with the use of relevant criteria.
Each discipline and development sector has specific methods that can be used to help gather
relevant information for technical monitoring purposes. For example, agriculturalists have ways to
assess soil quality, health practitioners have ways to test for the presence of health problems, and
conservation planners have ways of recording changes in the extent of a particular type of vegeta-
tion over time.
In this section, we aim to expand your repertoire of monitoring techniques by introducing tech-
niques that are useful for gathering information from and about people and the social contexts that
impact on their behaviour.
Criteria for selecting appropriate methods
e Timelines
Will you be able to gather the information within the necessary timeframe? If the method takes
too long to prepare, you will be unable to meet your reporting deadlines. If the method is very
time-consuming to carry out, you may not have enough staff to continue with project activities
as well as conduct the monitoring.
e Relevance
Will the method provide you with the information you are looking for? If the information
does not directly answer your information questions, you will have to reach your final answer
through guesswork! As this is not a strong basis for making important decisions about the fu-
ture direction of the project, it may be necessary to search for a more relevant method.
e Cost-effectiveness
Will you be using an appropriate level of resources to gather this information? Your monitoring
budget should be distributed appropriately across the range of monitoring activities. If a meth-
od is too time-consuming, or requires a large investment in training or specialised equipment, it
may not be as cost-effective as an alternative method to gather the same information.
There may be additional criteria that you can identify to help you select methods to use in your
project monitoring process.
It is important to keep in mind that project monitoring is done to meet the practical informa-
tion needs of the project. Practical, easy-to-use and reliable methods that will deliver the infor-
mation are preferable to complex ones designed to investigate research questions that deliver
little practical benefit to the project.
58
Most useful for information
Monitoring and evaluation: tools for biodiversity conservation and development projects
Techniques for gathering information
Other factors to consider when selecting
the most appropriate technique
Secondary
material and
documents
Individual
interviews
Group interviews
Key informant
interviews
Storytelling and
focus groups
Critical event
analysis
Participant
observation
Questionnaires
and surveys
Forms
Diaries and
logbooks
Photographs and
videos
Calendars
Daily and weekly
routines
Mapping
Ranking
Rich source of information that is readily
accessible.
Allows confidentiality. Time consuming but can
be a rich source of qualitative information.
Capture information from up to 10 people in a
short time. May not reveal sensitive information
but may be more detailed.
Useful for gathering objective views and
‘second-hand’ perspectives.
Useful for gathering information on a specific
topic—generates dialogue and reflection. Rich
in anecdotes that carry outcomes indicator
information.
Useful information for diagnostic studies
where a problem is being investigated or for
reconstructing the way it was solved.
Time consuming but useful for gathering
information about the way people behave when
going about their normal tasks without project
staff present.
Useful for collecting quantitative information
from a large number of people.
Useful for collecting information from a large
number of people at regular intervals, and that
can be used for comparative purposes.
Useful for collecting quantitative and qualitative
information that can be used for comparative
purposes.
Useful for recording changes over time for
comparative purposes.
Useful for identifying cyclical patterns or links
between behaviour and the seasons.
Useful for periodic, rather than regular,
collection of information using forms or
interviews.
Useful for collecting spatial information on the
indicators being monitored.
Useful for collecting information on priorities
or preferences where there are options
involved.
Monitoring and evaluation: tools for biodiversity conservation and development projects 59
Exercise: Methods for collecting information
. Think of a question, relating to your project’s target group, that could be discussed in an interview
situation.
Read the descriptions of the various types of interviews in ‘A toolbox of techniques for gathering
information’, including Critical Event Analysis.
. Consider the pros and cons of each interview method, relative to the question that you chose.
Use the summary table of ‘Techniques for gathering information’ to help you decide which would be
the most appropriate.
A toolbox of techniques for gathering information
e Secondary material and documents
Much of the information you seek may already exist in documents and this can save you from dupli-
cating work. For example:
= Project records—forms, logbooks, budgets and financial reports.
* Supervision checklists and reports.
=" Correspondence.
=" Media—articles published in newspapers and videos of television reports.
= Statistics, documents and reports published by other organisations, such as government depart-
ments and other NGOs, for their own use.
Secondary material can help you to see patterns and trends that you can use to inform the ques-
tions you might use in interviews and surveys. They can also help you to select other appropriate
methods to investigate particular issues in more depth.
e Individual interviews
The confidential nature of individual interviews often allows interviewees to discuss issues more
openly and they may reveal more than they would in a group setting.
The interview is best conducted in a semistructured way, with open-ended questions being used to
collect information, experiences, descriptions, opinions and perspectives. You can ask the interview-
ees to clarify their answers and to expand on interesting views. By using similar questions with a
range of interviewees, you can bring out a range of experiences and views about similar events.
e Group interviews
Group interviews allow you to gather information about the experience and perspectives of a
larger group of people than is possible with individual interviews. Such interviews are often ho-
mogenous—they bring together people who have had similar experiences, or are part of the same
group. However, groups should not be too large, perhaps no more than 10 people. If necessary,
subgroups can be formed.
Group interviews are less likely to reveal sensitive information than individual interviews. However,
interviewees may provide more detailed information as they interact with one another and check
with one another on facts. Group interviews may need careful preparation on the part of the
interviewer so as to ensure that all the members have a chance to express themselves during the
interview. If the interview is dominated by a small number of members, it may be possible to hold
informal conversations with quieter members at the end of the interview.
Monitoring and evaluation: tools for biodiversity conservation and development projects
e Key informant interviews
You can interview people who are specialists on relevant topics for their views on the way the
project is influencing its target group. Such informants can be more objective as they are not
personally involved in implementing the project or benefiting from it. Key informant interviews
are particularly appropriate if they can provide information and perspectives not available to the
project staff.
They can be asked to provide information on the experiences and perspectives of others, such
as the target groups, assuming that their contact with the target groups has allowed them to
witness what they are being asked to describe. It will be important to verify these ‘second-hand’
perspectives with information received ‘first-hand’ from project beneficiaries and target groups.
Open-ended questions can allow them to describe things that may not have occurred to the
interviewee to ask.
Storytelling and focus groups
These groups are convened for a detailed discussion on specific topics.A variation on group
interviews, they bring together people who have had particular experiences, or who have an
interest in the topic. The interviewer (who serves more as a facilitator) asks few questions, and
encourages participants to provide longer, more story-like, responses.As participants respond to
one another, a dialogue can develop. The facilitator can guide the discussion from the sidelines.
By asking reflective questions, the facilitator can help the group to explore patterns and themes.
Critical event analysis
This method is a way of focusing interviews with individuals or groups on particular events or
incidents. The purpose of doing so is to allow interviewees to reflect on as many of the ele-
ments and processes that they can describe, so as to create a richly textured understanding of
the event.
Participant observation
This method allows direct observation of events, processes, relationships and behaviour by the
researcher. It can be a useful way to verify whether the information provided through interviews
and surveys can be trusted or not. It can also be used to collect information to help design
interview and survey questions.
Observation is best carried out by someone unknown to the observed group. If a fieldworker
familiar to the group was to do this, his or her existing perspectives and pattern of interactions
with the group may prevent him or her from observing some significant things. The best insights
can often be obtained by a researcher who joins in tasks with the group, becoming a ‘participant
observer’.
Questionnaires and surveys
This is a structured way to obtain information from a large number of people.As specific ques-
tions are used each time, it is possible to gather quantitative information. It is not normally a
useful way for finding out about people’s concerns and perceptions, as it is not possible to ask
questions to check on the meaning of responses.
As the questions cannot be changed once the survey starts, it may be necessary to carry out
some pre-research (perhaps using secondary material and group interviews) to identify the
questions to ask. If the survey is being used to collect a large amount of information, it may be
appropriate to draw on professional assistance. For example, a university can help to design the
:
’
'
'
'
'
'
1
1
!
'
'
1
'
'
1
1
1
'
1
'
!
'
;
‘
'
‘
1
1
'
!
1
1
1
1
1
1
'
1
1
1
1
1
1
!
1
1
!
1
'
'
1
1
'
'
1
1
1
1
1
i
1
1
1
1
1
1
v
b
1
1
questions and analyse them properly. 1
‘
a
Monitoring and evaluation: tools for biodiversity conservation and development projects
It is important to identify people to be targeted by the questionnaire. Records, such as course
participant lists, can be used to conduct follow-up surveys on people who have attended semi-
nars and training events. If you want to get information from the wider population your project
is targeting, you will have to identify the best place to locate respondents. In a survey on child
health, your survey might get more informative responses if you interviewed women in the
market rather than at the clinic.
Surveys are especially useful in monitoring awareness and adoption of project outcomes.
Forms
Forms allow project participants and beneficiaries to collect information on pre-selected topics.
The contents of the form are based on selected indicators.At regular periods, the forms are
collected and analysed.
This can be a very participatory monitoring method, as it allows stakeholders to play a role
in designing the forms, collecting the information and analysing the forms. The information in
the forms can be supplemented by individual or group interviews to find out more about why
changes are taking place.
It is important to select a relevant time-frame for the forms, and to collect them at the agreed
intervals. Forms can be used on a daily, weekly or monthly basis. It may not be appropriate for
respondents to complete the forms on an ongoing basis. Rather, it may be more suitable for
monitoring purposes to use forms during agreed periods.
Diaries and logbooks
These documents allow participants to record events over time. They can be kept by individuals,
groups and by the project organisation. They can record events and facts, as well as responses
and opinions about what happened.
They are flexible tools that can be used in a focused way (such as recording information about
farming methods) or in a general way to explore themes. They can provide detailed and qualita-
tive information for monitoring purposes.
Participants who keep diaries and logs can come together periodically to analyse the informa-
tion they contain, and to discuss and analyse any significant changes. The information can be used
for comparative purposes, especially if the same participants maintain diaries throughout the
project period.
Photographs and video
A range of techniques can be used, ranging from digital cameras to video tapes, GIS (Geographic
Information Systems, which involve satellites), and aerial photography. The pictures and images
should be linked to indicators, and suitable intervals identified for taking pictures. The pictures
and any negatives should be stored in a safe place, and clearly labelled to avoid any confusion
with pictures taken at a later stage.
By taking pictures at regular intervals, changes can be identified and used to stimulate discus-
sions with stakeholders to analyse the changes. The changes can also inform questions used in
interviews and surveys. The discussion can also focus on what is not visible in the pictures, such
as the presence of women.
Calendars
This method can be used to gather information for distinct periods of time, allowing analysis
of changes in key indicators over time.As with diaries and logs, they can be kept by individuals
or groups. The information can be placed in the calendar by participants as soon as it becomes
available.As with other methods, discussion with participants can be used to analyse the infor-
mation in the calendar.
61
Oo
Ld
Monitoring and evaluation: tools for biodiversity conservation and development projects
More than one type of information can be collected in a calendar. This can make patterns appar-
ent, such as the relationship between household labour needs and seasonal agricultural activities.
Calendars can be maintained over a number of years, although most projects will be more
interested in information collected at weekly, monthly and seasonal intervals. Summaries can be
prepared for longer-term analysis, such as an impact study.
Daily and weekly routines
A variation of the calendar, this method is used to compare the daily or weekly routines of dif-
ferent groups of people and of seasonal changes in routine. It can be a useful way to monitor
whether the project is placing heavier demands on certain groups and whether project benefits
are fairly distributed.
This method can be carried out periodically, rather than on an ongoing basis. It will help to
ensure that the information does not swamp the analysers, and that participants continue to be
willing to provide the information to the project.
Mapping
Maps can be used to collect quantitative and qualitative information on the geographic distribu-
tion of indicators. Information can be recorded manually on aerial photographs or topocadastral
base maps and comments offered by participants can be written on the side of the map.
The method can be repeated periodically to assess whether there are any changes in the indica-
tors over time. The same map can be used again, with new information recorded in a different
colour, or a new map can be developed and compared with the original. Maps are an appropriate
way of gathering information about people’s perceptions of conditions in their community.
Ranking
Ranking provides a way for individuals or groups to prioritise a set of options. It can be a very
participative method for allowing participants to identify the importance of the options in rela-
tion to one another. This method can provide quantitative information, which can be supported
by qualitative information gathered through conversations and individual and group interviews.
The facilitator might identify the options by analysing interview or survey information. In such a
case, ranking can be used to verify the issues and learn about the importance attached to each
from the perspective of participants. The facilitator can also invite the group to identify the op-
tions and then to prioritise them.
Monitoring and evaluation: tools for biodiversity conservation and development projects 63
3.3 MANAGING THE INFORMATION GATHERING SYSTEM
The M&E system has to be managed, particularly where there are a number of people involved in
gathering information, so that:
# The quality of information is consistent.
* There is a common benchmark for measurement.
=" Information is captured in a form that is usable for evaluation purposes.
= It is stored in a place where it will not be damaged, destroyed or lost.
« There is a system for storing information that makes it possible to retrieve an item at any
time.
When the system is being designed, decisions have to be made about:
* Who gathers the information.
* How it is captured for monitoring purposes.
" Where it is stored.
OL
The value of a learning co-ordinator CO
Paula is the person who is responsible for learning and capacity-building within the project.The learning role she
plays feeds directly into planning, reviewing and reporting on the project’s achievements. For her to do this, she
is actively involved in gathering and storing information. She has information in files and on the computer. She is
also a meticulous note-taker. There are certain qualities that she brings to this job that make her so good at it.
She is organised, systematic and thorough. She will, for example, spend hours sifting through the information to
extract the ‘nuggets-—those gems of information that are embedded within the stories that people tell and the
reports that they write but that are not always immediately obvious. She is also good at asking questions so that
she can unearth these gems.
(Urban Sustainable Conservation Management Project)
Exercise: Finding the nuggets
Who, in your project, looks for the ‘nuggets’ in the information gathering process? Is this role recog-
nised and valued within the project?
Is there anything you could do to develop this role further? (This may require thinking through the
formal links between gathering information, and learning, reviewing, planning and reporting.)
S
The value of a database
External staff all feed information into the database, from a range of sites. It is a custom-designed database, on
the computer, that includes biodiversity and social data. Everyone can see it at any time and so there is no need
for internal reporting. The database is maintained by head office, there is a GIS technician who checks the spatial
data and makes sure they have been entered correctly. The data are then fed through to Scientific Services who
produce the quarterly and annual reports.
(Conservation Partnerships Programme)
64 Monitoring and evaluation: tools for biodiversity conservation and development projects
Ne ee Nee ata atte i ee el ee
It is likely that the gathering will be done by a number of project participants, some or all of whom
may not be based at the project office. It is also likely that at some stage the information will have
to be transferred to a central point in the project office, either in a ‘raw’ or ‘processed’ form, to be
assembled for evaluation purposes or stored as project records. ‘Processing’ could take the form of
entering the information into a computer database or writing it up in a standardised format.A flow
chart is a useful tool for depicting the flow of information in a monitoring system.
SOURCE | ¢ SOURCE 4
Q;> REPORT T# DONORS <2,
; aan vei
REPORT 10 REPORT TO %
TARGET GROUP COLLEAGUES
‘>> REPORT TO BiNEFICIARIES ‘D t
SOURCE 2 SOURCE 3
= me mee mee mee me DATA PROCESSING, ANALYSING AND EVALUATION
Two aspects of managing the information system—dquality and storage—will be discussed in more
detail here.
What makes the difference between good-quality and poor-quality information?
Information used for monitoring purposes must provide an accurate representation of what is
happening. This does not necessarily mean that it has to be 100% numerically accurate or ‘correct’.
What is important, is that:
= The whole picture is represented (e.g. that sampling is an accurate reflection of the whole).
= The potential for bias and ‘yea-saying’ is taken into account in the design of interviews, surveys
and questionnaires.
# ‘Interference’ or ‘noise’ is identified and avoided or filtered out of the information as far as pos-
sible (e.g. the effect of extraneous factors).
* Information that is going to be used for comparative purposes is comparable (the apples and
apples vs apples and pears syndrome).
Monitoring and evaluation: tools for biodiversity conservation and development projects 65
Most information gathering techniques need careful planning and design.
= Where there are a number of people gathering the same type of information, the use of forms
is a good way of achieving consistency. This can be done with a survey questionnaire, with care-
fully constructed closed questions.
= By comparison, a set of open-ended questions is typically prepared for interviews.
Why is a storage and retrieval system necessary?
Many of the monitoring techniques involve comparisons over time, or looking back at project
records to identify changes. The material that you collect throughout the project monitoring proc-
ess is valuable. It can be used for:
e Comparison purposes
If you collect similar information at regular intervals, you can compare older material with the
latest information. This can highlight changes over time that may not be evident when the infor-
mation is considered in isolation.
e Impact studies
The information and data that you generate during the life cycle of the project can be used as a
source of secondary material during an impact study. This information highlights shifts over time
that can help determine the effectiveness of the project’s outcomes.
e Future needs analyses
This information can provide insight into the groups that have benefited from the project, and
can help to identify other groups to be targeted by new project initiatives.
Storage systems have to safeguard the material from damage or destruction resulting from fire or
floods. If it is in electronic form, there should be a back-up stored off-site. Most importantly, the
information should be retrievable. There should also be a mechanism for tracking the whereabouts
of any original hard copies that have been temporarily removed from the storage system.
Exercise: What systems for information management does your project have?
Consider the following questions in relation to your project and create a ‘mind map’ of your project’s
storage system and its strengths and weaknesses.
|. In what form is your monitoring information stored?
Electronic files—text, spreadsheet, database, other?
Paper files—minutes and notes, forms, records of laboratory analysis?
Videos, tapes or CDs?
Maps?
Samples?
Any other form?
66 Monitoring and evaluation: tools for biodiversity conservation and development projects
. Where is it stored?
. Who is the person who knows most about where to find something? If he, she or you went on a
six-month sabbatical, would someone else be able to find the information needed for M&E?
. Who has access to the information?
. Is itin a form that is ready for M&E use or is further processing or analysis needed?
. If there was a fire in your office, would you be able to complete your next annual progress report?
. Do you have any recommendations on better management of your monitoring information?
InjSummany,
In Section 3 we:
* Identified that monitoring information can be outcome- or output-related and classified as
primary or secondary, qualitative or quantitative information.
* Introduced a number of methods for gathering monitoring information, drawn from the
social sciences and from participatory learning and action traditions.
= Explored some criteria that can be used to choose between methods.
* Described ways of managing material once it has been collected to ensure that it is useful
and accessible.
The focus of the M&E system should be on gathering information to enhance learning about
change and what makes change happen. Reliable methods of gathering information can have
an important influence on the quality and efficiency with which the information is collected.
However, the methods should not become an end in themselves—it would be better to select
a more suitable method than to continue collecting information of poor quality.
Care should be given to this step of the M&E process. It plays a critical role in enabling
participants in the development process to gather information of high quality that will deepen
their understanding of the project situation, allowing them to influence learning, decision-
making and accountability.
REFERENCES AND FURTHER READING
FEUERSTEIN, M.-T. 1986. Partners in evaluation: evaluating development and community programmes with participants.
Macmillan, London.
GUIJT, |. 1998. Participatory monitoring and impact assessment of sustainable agriculture initiatives: an introduction to
key elements. SARL Discussion Paper No. |. International Institute for Environment and Development, London.
http://www.iied.org/pubs/pdfs/6 1 391IED.pdf (accessed January 2008).
PRETTY, J., GUIJT, |. THOMPSON, J. & SCOONES, |. 1995. Participatory learning and action: a trainer’s guide. International
Institute for Environment and Development, London.
WADSWORTH, Y. 1997. Do it yourself social research, edn 2.Allen & Unwin, St Leonards.
Monitoring and evaluation: tools for biodiversity conservation and development projects 67
Section 4: ANALYSING THE INFORMATION
The process of analysing the information does not necessarily occur at a time and place different
from that of collecting the information. It occurs when you focus on two important questions:
# What are the significant differences between what we said we would achieve and what we are
actually achieving?
= What does this mean for the future of the project?
These questions will help you to make sense of the information that has been collected—to
understand what has and has not worked and the reasons for this. This lays the groundwork for
acting on whatever lessons are contained in all the information now at your disposal.
As various stakeholders may be involved in providing, collecting and analysing information, it would
be surprising if you do not face differences in and disagreements on how the information is under-
stood and interpreted.An important part of this process lies in acknowledging the various inter-
pretations, and allowing them to inform your own perceptions and understanding.
In analysing the information, you are trying to make sense of it so that you can make recommenda-
tions for appropriate action.
By the end of this section, you will have:
" Guidelines for assessing actual outcomes and outputs against expected outcomes and outputs.
=" Guidelines for assessing assumptions.
Explored the value of building shared understanding among different stakeholders and how this can
be done.
= An overview of different types of evaluations and evaluation methods.
= Been provided with a sample Terms of Reference for contracting external evaluators.
= A deeper understanding of how social development evaluators work.
68 Monitoring and evaluation: tools for biodiversity conservation and development projects
4.1 ESTABLISHING IF THERE ARE SIGNIFICANT DIFFERENCES
This first step in this process is partly descriptive and partly comparative. Its purpose is to identify
what was supposed to be happening at this stage of the project (this information will be in your
plan) and to compare it with what is actually happening. The questions you will be seeking to an-
swer are:
« Are there any differences between what was planned and what was actually achieved? In other
words:
» Did the project accomplish more than what was expected?
» Did the project achieve less than what was expected?
As the real world is a dynamic, ever-changing environment, it is highly unlikely that the actual situ-
ation in which you find yourself will be identical to the intended situation. It will probably not be
helpful to describe all the differences between the actual and intended situations, as some of the
changes may be relatively small and of little importance.
However, it will be important that you describe any significant changes. The significant changes are
interesting because they will form the focus for your analysis and subsequent action. The changes
can be identified by monitoring the indicators at the level of outcomes and outputs, as well as by
monitoring the enabling factors or assumptions. (These are components of the project plan. The
definitions are provided in Section |.)
There is usually one guiding question for each level of indicator.
e At the outcomes level, the guiding question is:
* Is the project having any positive influence on the target group, beneficiaries and/or environ-
ment?
This question focuses us on outcomes at the level of Development Goal and Project Purpose.
The indicators you have developed for your outcomes will help you assess the degree of
change that has taken place. Project monitoring activities are likely to focus on the intermedi-
ate targets that you have established, as this will allow you to review the changes in the project
environment over the life of the project.
e At the outputs level, the guiding question is:
* Is the project making progress towards its objectives and how efficiently is it using resourc-
es?
Here you are looking for information about the project’s progress in implementing its planned
Activities and achieving its key Outputs. It is important to pay attention not only to what has
been achieved, but also to the quality and the scale of the achievement.
The focus of output indicator monitoring, especially at the start of the project, will be on
progress towards implementing the Activities described in the project plan. The Activities are
the easiest part of the plan to change in response to problems and unexpected events. By
monitoring the progress of the Activities, you are able to respond quickly to emerging issues
before they escalate into larger problems.
Monitoring and evaluation: tools for biodiversity conservation and development projects 69
You can develop a false sense of security in the status of the project if you simply monitor its
progress towards its goals. There are many projects that have made a good start with progress
towards their results, but have run out of resources before the end.As a consequence they have
struggled to complete all their planned activities. By paying attention to issues of resource use and
efficiency from the start of the project, you will help to ensure that your project will have sufficient
resources to achieve all its objectives.
The two main resources over which the project has some measure of control are staff time and
finances. Both of these can be viewed as ‘consumables’-—once they have been used for one thing,
they cannot be used for something else.
Staff make efficient use of their time when:
» They use as much time as was scheduled to complete planned activities.
» There is an appropriate number of people working to complete a task.
» They are doing work appropriate to their skills and experience.
Efficient use is made of financial resources when:
» The budgeted resources are used to complete a task.
» There are few quality problems that require tasks to be repeated.
» Appropriate savings are made on unnecessary expenses.
Members of the project team should aim to monitor and control costs as they carry out their
work. This is more likely to happen if they are clear about the resources that are available to them.
By developing an activity-based budget for the project, and providing regular (monthly) financial
reports, it is easier for staff to monitor and control expenses.
Establishing the nature of the difference
Are there significant differences between what we said we would achieve and what we are actually
achieving?
The monitoring information may show that the expected changes are taking place. If so, this affirms
that the project intervention is appropriate in bringing about the desired changes.
However, if you find that there is a significant difference between what you expected and what is
actually taking place, the situation requires further investigation. It is under these circumstances
that the views of an external evaluator can be particularly helpful. The aim of this stage of analysis
is to deepen your understanding of the situation and to inform the recommendations you might
make for responding to this unexpected development. This is achieved by asking the questions:
= Did we achieve more than we expected?
= Did we achieve less than we expected?
Ss
70
Monitoring and evaluation: tools for biodiversity conservation and development projects
Did we achieve more than we expected?
Significant changes in what you find will not always be for the worse. There might be occasions
when you find that there is a greater response than you expected, or a positive outcome that
was not foreseen at the planning stage.
In the original plan, we were going to appoint negotiators. But we found that landowners were willing to
negotiate with conservation staff, and so we decided not to appoint negotiators, but to use extension staff
instead.
There is also much skills development going on but we are not consciously monitoring it.
(Conservation Partnerships Programme)
It might be important to understand why this has happened, as it may help you to identify how
you can use this greater response to the project’s benefit. It may also help you to have a better
understanding of the conditions under which you are working.
Some questions that might guide your investigation into achieving more than you expected at
the outcomes level are:
* Is the increased response a result of the project’s work, or is some other factor responsi-
ble?
* Is the response likely to place greater demand on the project’s services in the next period
of implementation?
= Are the project’s targets sufficiently ambitious? Should the objectives be stretched further
to reflect this new reality?
When investigating higher-than-expected achievements at the outputs level, you are looking
for information on why it has happened, and whether there are any future implications for the
project:
# What is driving the increased demand?
= Can the project accommodate it?
" If so, can you adjust budgets and plans to meet the raised targets?
= If not, can you work with other organisations to pick up the surplus demand?
= Can you reduce expectations in any way?
You may find that you can complete tasks in less time than you anticipated, and that there are
less expensive ways to obtain certain services from various suppliers. When you make faster
progress or better use of resources than you expected, you have the opportunity to re-allocate
either the time or resources to other activities—as long as you are monitoring the situation
and become aware of it early enough.
Did we achieve less than we expected?
If monitoring reveals that the rate of implementation is slow, with few benefits materialising,
it may suggest that the project is not following an intervention strategy appropriate to the
context.
Monitoring and evaluation: tools for biodiversity conservation and development projects 7\
One of the purpose-level indicators for the pilot project was the adoption of contractual conservation agreements
by landowners. By the end of the pilot, no contracts had actually been signed, although several were pending. We
had not fully understood the complexity of land ownership.
(Conservation Partnerships Programme)
Further investigation will be needed to understand why the project is not progressing as
planned.
At the level of outcomes, the investigation may reveal that:
# While the intervention strategy may have been appropriate when the project was first
planned, changed conditions mean that it is no longer so.
« The project was planned with insufficient information and a limited understanding of the
context, resulting in an inappropriate or over-ambitious intervention.
= There might be problems with the quality and timing of the project’s work.
If outputs monitoring shows that there is a slower rate of implementation than expected,
that there are significant delays in starting or completing certain activities, then you are facing
progress problems. There could be many reasons, ranging from staff issues to external factors.
These problems with progress can represent a significant threat to the success of the project
and so it is important to identify them at an early stage when it may still be possible to prepare
a response that can be quickly and easily implemented.
Over-spending or poor use of resources can be a rich source of learning for the project:
= Costs may have risen since the budget was prepared. This can be a timely warning for revi-
sion of the budget.
= Over-spending may be due to poor financial management, such as taking on additional
activities. The budgets might have to be adjusted to make up for the additional expenses by
reducing the finances available to other activities.
Two useful resources on financial management include: Shapiro (1995, Financial management for self-
reliance) and Randel (2001, Planning for implementation).
At the assumptions level, the guiding question is:
= Are the enabling factors present or are they absent?
By monitoring the enabling factors that will contribute to project success, you are not looking for
progress or reaction. You are assessing the presence, or the absence, of these critical enabling fac-
tors.
The external factors, by definition, are beyond the control of the project. The reason for including
the enabling factors in the monitoring process is to provide an ‘early warning system’ that can draw
the attention of the project team to potential risks to project success.
72 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Are the enabling factors present?
Enabling factors are also known as assumptions. By definition, their potential influence on the
objectives is unknown when the project is planned. The assumption of the planners is that the
enabling factors will be present and will positively support the implementation of the project.
Understanding and support for the stewardship concept has grown remarkably—much faster and more
widespread than we had anticipated. The willingness of landowners to negotiate with conservation staff was
underestimated.
(Conservation Partnerships Programme)
As you gather information about the enabling factors, you may find that the assumption has
proved true. In the example above, the project developer is referring to the assumption that
the project would receive the support of the target group. This is important feedback for the
project team. It means that the project strategy continues to be valid, as one of the critical
enabling factors is supporting the project’s work.As there is little significant difference between
what was expected and what the monitoring information is revealing, the project does not have
to respond in any way.
If your enabling factors are in place as expected, you will probably have to continue to moni-
tor them at regular intervals for the rest of the project period. The external world is a dynamic
place, and a change in the status of the enabling factors could have many causes.
The strong appearance of an enabling factor that significantly exceeds planned assumptions is
not necessarily a good development for the project. The project team may be tempted to aban-
don the project plan and to revise the strategy, taking the new situation into account. While
conditions may be supportive at present, it may not be possible to predict how long this will
continue, and whether it is sustainable.
Great care should be taken when deciding whether or not to increase the ambition of the
project strategy and objectives because of such changes. There may be times when it will be
beneficial to stretch the targets set for the objectives in response to a positive development.
But there may be other times when it might be better to maintain the project objectives at
their original level of ambition.
e Are the enabling factors absent?
As the enabling factors are outside the control of the project, it is not possible to know when
they will come and when they might depart. Ongoing monitoring of the enabling factors can
help to identify the time when a factor changes its presence, and thus its influence on the
project.
There are a number of conflicting and confusing policies that are coming out of the different national
departments. This has made it difficult to secure water-tight legal agreements with the landowners.
(Conservation Partnerships Programme)
Monitoring and evaluation: tools for biodiversity conservation and development projects 73
By waiting to see what the longer-term trends will be, and whether there are short-term fluc-
tuations in the status of the enabling factors, the project can identify the way it should respond.
Before it decides to make any changes in the project plan, it might be worth investigating the
forces that may be influencing the enabling factors. If these forces and systems can be positively
influenced, the enabling factors may become more consistent, to the project’s benefit.
Should the enabling factors continue to fluctuate, or be absent, the project’s success is threat-
ened. It will be appropriate to investigate the situation more closely and to start a problem-
solving process to protect the work that has gone into the project.
Exercise: A snapshot view of your project's progress
Taking into account the stage of implementation of your project, consider the following questions:
éS
In which areas of delivery has the project achieved more than it said it would? Where has it
achieved less than expected? What are some of the reasons for the difference?
Which assumptions are you monitoring? Have there been any changes in these external conditions
that are critical to the success of the project? Are there any assumptions that should be changed,
added or dropped?
74 Monitoring and evaluation: tools for biodiversity conservation and development projects
4.2 CLARIFYING AND AGREEING ON THE REASONS FOR ANY DIFFERENCES
The purpose of M&E is to help you review your work in progress and to see whether or not you
have to make any changes to the way you are implementing the plan. If you find that there are
some significant differences between what has taken place and what was planned, you are likely to
want to respond to them in some way.
Before you act, though, it is important to be clear on what you are responding to. While it is some-
times possible to spot the reasons for why your plan did not proceed as intended, the members of
your project team and other stakeholders may hold different opinions on these underlying causes.
This step is important, as it will help you to clarify the reasons for the differences and also to build
agreement among important stakeholders about these reasons. This is an important foundation for
what will follow—acting on this new insight and understanding.
Differences in outcomes and outputs can be ascribed to external or internal factors. It is helpful
to make this distinction as a starting point. Enabling factors are almost always external. It is also
useful to probe beyond what appears to be the ‘obvious’ reason for a mismatch between what was
planned and what actually happened in case there are deeper underlying causes. The picture that
is being conveyed by the M&E information may be being influenced by, for example, a bias in the
selection of indicators, which would start to become apparent at this stage through a process of
interrogation and clarification.
Some of the tools that may assist you in clarifying and reaching consensus on the reasons for differ-
ences, and what this might mean for the project, are discussed here.
One of the challenges in analysing the information that you are collecting on the project’s out-
comes and outputs, is to build a shared understanding among the different stakeholders about the
meaning of this.
If stakeholders in the project do not agree with one another on the causes for why things turned
out differently from what was expected, it will be difficult to get support for any changes that you
may propose to the way the project is carried out.
Shared understanding forms the basis for joint ownership of the analysis and for getting support
for any changes in the project’s approach.
Monitoring and evaluation: tools for biodiversity conservation and development projects 7S
The process of building a shared understanding should not try to suppress the different opinions that
people may hold because of their differing experiences and mental models. Rather, a dialogue on what
is happening, and why, allows participants to share their perspectives, knowing that their colleagues will
listen to them.
In such a dialogue:
= People are expected to challenge one another and the team’s mental models (more information
on this is presented in the next section).
" Shared understanding will emerge as the team makes sense of the conflicting perspectives people
hold.
= Consensus may not result, but people will have a better appreciation for the reasoning others are
using, which can lead to an appreciation for the recommendations for changes that will be made.
Participants in the dialogue can range from two people to the whole team, and it can include stakehold-
ers from outside the project, such as target group representatives.
Reference: Raelin (2000, Work-based learning)
As we review the monitoring information, we may find that we did not meet the desired results be-
cause of the way in which we carried out the work. There may have been problems with the quality of
our work. Perhaps our planning and assumptions about what was needed were incorrect from the start.
Often, there can be pressure to overlook these causes and to look for other reasons for why things did
not go as planned. It can be uncomfortable to find the fault lies in the way the work was carried out or
managed, especially if senior people in the organisation were involved.
The pressure to pass over poor management and capacity problems can be stronger in organisations
that receive donor funding. If staff are concerned that the organisation will not receive further funding
because of its poor performance, they may be reluctant to reveal the underlying causes of the prob-
lems.As a result, the project and the organisation are less likely to learn from these experiences and to
develop ways to improve their ways of working.
For staff to identify and discuss openly any internal, organisational problems that affect the project's re-
sults, the project team should support problem-solving and reflective discussion. In such a work climate:
=" Team members are trusted and supported by their colleagues.
= Staff are encouraged to take responsibility for the consequences of their work.
= Problems and mistakes serve as a source for reflective learning, not for blame.
»))
When we had our last evaluation, we decided that we would be honest and open in sharing the findings za
with others. Obviously, there were things that were and were not working about the project.We dared «!
to say that things were not perfect, that we had got some things wrong. We shared this with our Project
Advisory Group and with the C.A.PE. Implementation Committee. | think this opened people up to the
possibilities of what an evaluation can do. Our donors were also very supportive and impressed. So were
the environmental educators. | think people were able to learn through our experience.
i
i]
|
|
a
a
(Urban Sustainable Conservation Management Project) 1
if
i
(Senge 1994) .
t
76 Monitoring and evaluation: tools for biodiversity conservation and development projects
4.3 CONDUCTING FORMAL EVALUATIONS
Formal evaluations have a useful role to play in analysing information about the progress of the
project. Much of this evaluation work can be done by internal project staff. However, another way
of bringing new insights and perspectives to these questions is to make use of external evaluators.
These evaluators can bring certain skills and expertise that are not necessarily located within the
project.
There are different types of evaluations, as well as different evaluation methods, that can be drawn
on. The selection of the most appropriate type and method will depend on the project particulars:
the stage in the project life cycle and the type of activities being monitored.
Types of evaluations
Specific types of evaluation have been developed to address the various evaluation questions that are
relevant at different points in the project cycle.
e Ex-ante
An assessment of the project plan and approach, before it starts, to assess whether or not it has a
good chance of achieving its objectives.
e Midterm
A review, carried out midway through the life of the project to determine if the project is likely
to be successful and to assess whether or not any changes in approach should be made to ensure
SUCCESS.
e Terminal
An evaluation at the close of the project to determine if it has accomplished its objectives.
e Impact
An assessment of whether or not any benefits resulting from the project have led to a sustained
improvement in the issues it was addressing.
One final distinction in monitoring and evaluation terminology should be noted.
e Formative evaluation
Evaluation that is ongoing, or that provides continuous feedback to inform ongoing changes and
improvements in a service or a product.
e Summative evaluation
Evaluation that is done periodically to inform reporting and decision-making. The decisions are
typically at a high level, such as whether or not to continue funding for a project.
In almost every situation, monitoring activities provide formative evaluation of the project’s work. Cer-
tain evaluation activities, such as a midterm evaluation, are also formative, as they provide feedback that
is used to make changes in the project’s work. Other evaluation activities, such as the impact evaluation,
are summative in purpose, as they are used to inform decisions on whether or not to continue with the
project.
Both formative and summative evaluations allow us to ask, ‘What can we learn from this experience
that will help us tomorrow, or the next time, or may be relevant in other places?’ Formative evaluations
answer this question in ways that directly inform project implementation, while summative evaluations
provide insights to those further away from the work, such as donors and host programmes.
Monitoring and evaluation: tools for biodiversity conservation and development projects 77
Particular methods that are useful for evaluation purposes include:
e ‘Before and after’ studies
These studies are based on baseline information. Subsequent assessment compares the current
status with the status at the starting point.
e ‘With and without’ studies
Assessments are made of both people who adopt and those who have not adopted the project’s
services. The study aims to assess whether or not the project is making any positive difference
among those who are adopters. The nonadopters can provide information on their awareness of the
project’s services, and why they have not adopted them.
e Longitudinal studies
An ongoing assessment is made with individuals or groups, selected as representative of the wider
target and beneficiary population. Changes in this sample group are assessed periodically. The group
can include both adopters and nonadopters.
The process of contracting the evaluators has to be well thought through and rigorous—both in
terms of the legal requirements and the objectives of the evaluation. Clear Terms of Reference
(TOR) are an important starting place to ensure that you get the most out of this process.
Furthermore, different evaluations call for different ways of working. Status assessments, for exam-
ple, require a specific set of skills and expertise, as do ElAs. One of the challenges facing projects
in the biodiversity conservation sector, is how to assess the progress and impact of their social
objectives. Social development evaluators bring a different approach and set of skills to their work.
The table that follows the sample TOR outlines some of the principles and practices that typically
inform the work of social development practitioners.
78 Monitoring and evaluation: tools for biodiversity conservation and development projects
SAMPLE TERMS OF REFERENCE FOR EXTERNAL EVALUATORS
(FROM CAPE FLATS NATURE)
CFN began in June 2002 as a partnership between the
City of Cape Town, the South African National Biodiversity
Institute, the Table Mountain Fund (WWF-SA) and the
Botanical Society of South Africa. Cape Nature and the
Table Mountain National Park joined the partnership
in 2006.The project’s purpose is to build good practice
in sustainable management of City nature conservation
sites in a way that benefits surrounding communities,
particularly those where incomes are low and living
conditions poor.
CFN was initiated with seed funding from the City
and the Table Mountain Fund (TMF), as set out in a
three-year contract between the City and TMF on the
one hand, and the South African National Biodiversity
Institute (SANBI) on the other hand as the implementing
partner. The completion of this contract at the end of
June 2005 formally brought to an end the demonstration
phase of the project, and called for a major evaluation.
The evaluation was conducted by an independent
external evaluator.
As part of the demonstration phase, CFN raised
additional funds from the Critical Ecosystem Partnership
Fund (CEPF).This required the drafting of a logical
framework, which added further content to the City/TMF
contract, and also extended the terms of reference of
the project to design a replication and roll-out strategy.
The 2005 evaluation constituted a mid-term review
for the CEPF-funded project, and its recommendations
contributed to the development of the replication and
roll-out strategy. The CEPF funding comes to an end in
September 2007, and the contract with CEPF requires a
further evaluation to be completed.
CFN again requires the services of an independent
external evaluator to facilitate, conduct and document
this evaluation.
Objectives
The objectives of the evaluation are as follows:
= Comprehensively document developments in the his-
tory, strategic approach and organisational processes
of the project since the previous evaluation.
= Assess the extent to which the project has met the
objectives of its contract with the CEPF, paying particu-
lar attention to the sustainability (including financial
sustainability) aspects of the project.
" Assess the relevance of the outputs to the purpose of
the project as set out in the CEPF logical framework.
# Strengthen the project by providing external perspec-
tives, insights and, where appropriate, recommenda-
tions based on broad developmental experience.
" Develop a draft framework for phased levels of sup-
port in establishing new sites, and for building and
maintaining civil society support for conservation,
particularly in low-income communities.
= Develop an outline and a process for the development
of a handbook on integrated urban nature conserva-
tion management to be completed by 2009.
= Draw and document lessons learnt from the develop-
ment of the replication and roll-out strategy in a way
that contributes to the implementation of this strategy.
" Build the capacity of the City’s Nature Conservation
Branch to write up case studies of their work in a way
that contributes to spreading a people-centred nature
conservation practice by documenting and sharing
lessons into the future.
" Develop the ability of Cape Flats Nature’s Capacity
Building Manager to build capacity of other partners,
e.g. Cape Nature to write up case studies of their
work in a way that contributes to spreading a people-
centred nature conservation practice by documenting
and sharing lessons into the future.
Method
The method used should facilitate organisational
learning, at the same time enabling accountability to
funders. It should integrate the evaluation process into
the organisational life of the project and that of the City
Nature Conservation Branch as far as possible to avoid
creating additional work for the project team and its key
partner, and to ensure support for the implementation of
recommendations that emerge.
An effort should be made to use both qualitative and
quantitative methods of assessment as far as possible,
and recommendations for future information gathering
will be made where the necessary information is not
available.
T
Monitoring and evaluation: tools for biodiversity conservation and development projects 79
The Project Advisory Group and project team should be
involved in finalising the contract and evaluation process
with the external evaluator.
CFN will arrange meetings for the process, e.g. the
Project Advisory Group and Champions Forum, but the
consultant will set up individual interviews.
Outputs
" A City Nature Conservation Branch capacity-building
workshop on writing up case studies.
A report that synthesises all materials and the
outcome of interviews, and develops an independent
perspective and interpretation. It should conclude with
concrete recommendations covering all areas of the
TOR.
A draft framework for phased levels of support in
establishing new sites, and for building and maintain-
ing civil society support for conservation, particularly in
low-income communities.
An outline and a process for the development of a
handbook on integrated urban nature conservation
management.
Reporting and management
CFN’s Project Manager will manage the implementation
of the evaluation, and the consultant will report to her.
Timeframes
Proposed timeframes, to be finalised in consultation with
the evaluators and the City Nature Conservation Branch
(for the case study workshop), are as follows:
® The finalisation of the consultant contract and evalu-
ation process will take place in time for a Project
Advisory Group meeting in May, to form part of the
process.
" The case study workshop will take place in May, with
case studies due in draft by early June, and finalised
by the end of June.
= The Champions’ Forum (drawing together community
champions from the four pilot sites to share experi-
ences and draw lessons) on 2! June may play a role
in the evaluation.
= A draft report will be submitted by the end of July for
comment, and the report will be finalised by the end
of August.
Reference documents
Project materials to be consulted in framing the
evaluation include:
= CEPF logical framework.
= City/TMF contracts.
= Reports to funders and funder comments on these
reports: biannual narrative reports to CEPF and Phase
2 reports to City.
= Soal, S. & Van Blerk, R. 2005. Report to Cape Flats
Nature on the outcome of an evaluation.
Roll-out strategy outputs.
Financial strategy.
# Memorandum of Understanding.
Hill, S. 2006. Networking people and nature in the
city.
= Videos: Cape Flats Nature: biodiversity for the people;
Bridging the gap: new conservators speak; Networking
people and nature in the city.
Case study file.
Press clips file.
80 Monitoring and evaluation: tools for biodiversity conservation and development projects
Principles underpinning a social development approach to evaluation (extract-
ed from CDRA's website)
e Learning and accountability
External evaluation should enable both learning and accountability. Learning is to search out
understanding, insight, knowledge or skill through reflecting on experience. Accountability is a
transparent story of one’s successes and failures. When working with small-scale social change
(where work occurs in the realm of the invisible, personal and interpersonal), organisational learn-
ing is the most reliable route into external accountability. In CDRA, we have discovered that when
we share our own field experiences for the purpose of learning and improving future practice, ac-
countability happens. It flows naturally. But we also know that when the spotlight is on ‘Account-
ability’ (with a capital ‘A’), the liberty to be vulnerable shrivels, and the need to prove success
or failure dominates. The outcome? Superficial learning and accountability; a neat report for the
donor;a tick in the manager’s to-do column.
Quantitative and qualitative measures
In this approach to evaluation, quantitative and qualitative measures are combined. Ideally, if or-
ganisations have mechanisms for feedback and data collection, basic quantitative information on
‘outputs’ is available as a starting point for the evaluation. If not, the evaluation has to begin with
generating some of the required information, and if time allows, recommend systems for future
use. However, where feedback on ‘quality’ and strategic appropriateness is sought and where
different strategies are valued in different ways, quantitative measures are insufficient. Qualitative
measures and an approach that draws on the understanding of all involved are far more effective
in providing a meaningful indication of impact and value.
Measuring impact—evaluation or research?
Measuring the impact of developmental interventions is critical to the process of learning and ac-
counting. It is possible and necessary to collaboratively measure impact, quantitatively and quali-
tatively, on those directly involved in and directly impacted by the intervention, and from which
one can strategically argue or anticipate the likely contribution of that intervention to change
downstream'. But measuring or proving impact downstream, scientifically for governments, on the
ultimate ‘beneficiaries’ and wider, is considerably more complex and not a task that practitioners
can be expected to take on as part of their direct evaluations in practice.
Practitioners may be able to demonstrate and illustrate impact downstream through following
up particular case studies, but scientifically measuring impact downstream requires developmen-
tal research, requiring skills, resources and time that they do not normally have. Donors have to
fund research for this impact on ultimate and wider ‘beneficiaries’, relieving the pressure on M&E
systems, but contributing the research to their learning and enabling practitioners to focus on
evaluation for practice development.
Participation
Essentially, evaluation for learning does not substitute for facts and figures—but goes beyond it.
It is quite literally ‘formative’ as participants’ understanding of their work or experience is deep-
ened—formed—through the process of evaluation.An evaluation ‘finding’ is arrived at through a
form of structured conversation, a process of questioning, listening, making meaning and deepen-
ing understanding. This kind of ‘finding’ is both a valuable and valid one. It builds on the logic of
‘doing is knowing’, and validates the knowledge generated by considered experience.
' Outcomes mapping
Monitoring and evaluation: tools for biodiversity conservation and development projects
The central and most important source of information in developmental external evaluation is
people themselves. Their perceptions (be they reflective of a majority viewpoint, or not) are valid
material for the evaluation. The processes being evaluated are human processes, therefore the
measures must be human too. Seen in this way, a ‘perception’ of impact is a form of impact in
itself.
The external evaluator’s role is to create an environment that is conducive to real engagement.
Engagement in self-reflection, group exchanges and in one-on-one interviews. It is his/her role to
help participants listen, to themselves and others, as both a form of accepting and of asking why.
It is his/her role to help participants to view their scenarios from different perspectives, and to
spark their dynamic for change. The evaluator is required to sink deeply into each perspective be-
ing offered, yet retain a critical and independent stance. This is what frees her/him to incorporate
all contributions towards providing a whole picture of the organisation’s functioning and impact.
Internal and external foci
Exploration for the reasons behind things working, or not working, requires bringing the internal
organisational aspects into play with the external factors.
The internal aspect of evaluation is helpful because it addresses those things that can be changed
or improved upon. It frames both problems and solutions in such a way that they are within the
reach of the organisation.
An evaluation addressing only the external might conclude that an organisation failed simply be-
cause of a shift in the external environment. However, viewed from the inside out, the evaluation
could conclude very differently: perhaps that the organisation did not have adequate environmen-
tal scanning systems...or that it never reviewed its strategies...or that its leaders were not fulfilling
their duties...or that there was poor team work.
You can do something about setting up effective scanning systems. You can do something about
that struggling leader. But you cannot control the outside world. In other words, if you make space
to consider the internal questions alongside the external ones, the conclusions will be different
and they will hold insights that are organisationally do-able.
Strategic connection
Finally, evaluations should be conducted with an eye on the future. If you want the evaluation to
be of real use to your organisation, the implications of all the findings and potential solutions must
be teased out. So, team work is poor. How might things be done better? What has to change?
What would that mean for individuals and for the organisation?
Final recommendations should incorporate the outcome of this reflection and learning and specify
the subsequent ‘ideal’ goals (strategic and organisational), as well as the processes and paths to be
followed to begin reaching those ideals.
(Soal et al. 2007)
8|
82 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Drawing up a terms of reference document
Taking into account the different types of evaluations, is it time for you to conduct a formal evalua-
tion of the project?
If yes, consider the following questions:
What is the purpose of the evaluation?
What level of objectives are being evaluated?
What are the intended outputs of the evaluation?
Who will manage the evaluation process?
Will it be an internal or external evaluation? Or a combination of both?
Draw up a draft terms of reference document.
In Summary,
In Section 4 we:
* Looked at the process of comparing actual achievements against planned achievements and
how this applies to outcomes, outputs and enabling factors.
* Introduced tools to assist the project team and stakeholders to find common ground on
the reasons for significant differences between what was planned and what monitoring
reveals as having been achieved.
= Explored the value of formal evaluations and, in particular, how social development practi-
tioners would approach an evaluation.
This stage of the M&E process is critical, as it forms the bridge between reflection on the
events and experiences of the past, and future action. It does this by investigating the signifi-
cance of what has happened.
Section 5 takes this one step further by looking at the way to identify recommendations and
the decision-making process for acting on the analysis.
REFERENCES AND FURTHER READING
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ.
RANDEL, M. 2002. Planning for monitoring and evaluation. Olive Publications, Durban.
SENGE, P. 1994. The fifth discipline fieldbook. Nicholas Brealey, London.
SHAPIRO, J. 1995. Financial management for self-reliance: a manual on managing the finances of a non-profit organisation.
Olive Publications, Durban.
SOAL, S., HILL, S. & REELER, D. 2007. Evaluation: a developmental approach. http://www.cdra.org.za (accessed January
2008).
Monitoring and evaluation: tools for biodiversity conservation and development projects 83
Section 5: ACTING ON THE ANALYSIS
There are two factors that are important for taking action once you have a good analysis of the
information you have collected: drawing out recommendations and effective decision-making. The
value of a formal evaluation (whether it is conducted internally or with the support of external
consultants) is that it helps to create the time and space for these ideas to be identified and fur-
ther explored. There are some broad categories into which recommendations typically fall and
which can be used to think through the future of the project, even in the absence of a formal evalu-
ation process. They are discussed below.
The likelihood of new actions being adopted and real changes being introduced into the project
relies on effective decision-making. In particular, it is important to consider who should be part of
the decision-making process, and what is being decided.
By the end of this section, you will have:
# A framework for identifying different types of recommendations.
= Explored different components of the decision-making process.
= Looked at how decision-making occurs in groups.
fl
1
1
i
:
i
i
1
1
!
® Been introduced to the notion of ‘resistance’. :
a
Monitoring and evaluation: tools for biodiversity conservation and development projects
9.1 DRAWING OUT RECOMMENDATIONS
In this step, we start to shift our focus from past experiences towards future actions. This involves
responding to any significant changes in the project and its environment by:
* Developing specific recommendations for follow-up action. by project staff.
* Identifying general lessons from this experience that might be relevant in other contexts.
Applying insights from one part of the project to other parts of your work can help you to avoid
encountering the same problems in the future. This can also help you to respond to the changing
context in which you now find yourself, allowing changes to the project plans that take account of
your environment.
Now that you are aware of any significant changes, and understand why they have occurred, this
part of the project monitoring process allows you to ask:
= Are there any significant consequences for your project?
* Should you do anything differently in future?
* What specific recommendations do you have for decision-makers?
Any recommendations you develop should be viewed as experiments. You do not know if the
proposed ‘fix’ will achieve the desired result and what will happen. But by adopting an experimen-
tal attitude as you consider possible responses and develop your recommendations, rather than
‘we must get this right’, you encourage innovation and risk-taking. This can help you to identify
responses that you might not normally have considered, allowing you to accomplish more than you
had considered possible.
identifying recommendations
Now that you have a better idea of what is happening, and why, you may want to respond in an ap-
propriate way. Your response may help keep the project on track towards its objectives, or it may
bring about a change in the project’s objectives.
By drawing on the lessons you have identified, you may identify practical steps you can take in re-
sponse to the changing environment. Should you:
= Keep doing?
» Do more of or less of...
» Do in different locations...
= Stop doing?
= Start doing?
What kinds of changes can you recommend?
Recommendations for change can be viewed as a process of ‘steering’ the project around any iden-
tified obstacles that might prevent it from reaching its destination. If any changes are needed, they
should be made in proportion to the obstacles you are trying to avoid.
Monitoring and evaluation: tools for biodiversity conservation and development projects 85
One helpful way to decide on steering changes, is to consider the following four options:
e No intervention
No response will be necessary when there is no significant change between the planned and
the actual status of the project.
Three, instead of four, consultative meetings were held with the landowners. They are in agreement with the =
proposed plan of action regarding the removal of alien vegetation on their land. ~/)
(Source: Project Developers’ Forum, |8—19 March 2008)
e Corrective action
This is appropriate when there is a difference between the planned and the actual status that
can be corrected relatively easily by:
= Adjusting or repeating certain activities.
or
# Re-allocating inputs (such as financial resources and staff).
The project was unable to secure meetings with the community representatives in the time allocated. The Zz
remoteness of these communities and the bad roads contributed to this. Meetings have been re-scheduled for \G
the next quarter.
(Source: Project Developers’ Forum, |8—19 March 2008)
e Re-plan or adjust the plans
This change can be introduced when there is a significant deviation from the planned status that
cannot be corrected immediately. Responding to the deviation may require making changes in
the project plan at the levels of:
= Development Goal or Project Purpose objectives.
= Outputs and Activities.
= Inputs.
One of the communities with whom the project is working is tied up in a complicated land claim. Instead of
asking community members to dedicate all their time to maintenance of hiking trails, the project has re-allocated
some of this time to supporting community reps in understanding their rights and the legal processes they have
to follow.
(Source: Project Developers’ Forum, |8—19 March 2008)
e Discontinue the project or individual project components
This extreme action may be necessary when the monitoring shows that the project’s objectives
cannot be achieved with the current intervention strategy and resources.
86 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Steering the project—different options
Look back at the various phases of the project in which you are involved. Can you find examples from
your own experience of the four categories of recommendations?
No intervention.
Corrective action.
Re-plan or adjust the plans.
Discontinue the project or individual project components.
What were your reasons (or thinking) for taking these different courses of action? Knowing what you
know now, would you take a different course of action?
Monitoring and evaluation: tools for biodiversity conservation and development projects 87
J.2 EFFECTIVE DECISION-MAKING
A key purpose of learning is to inform effective decision-making by providing timely and relevant
analysis. The decisions that are made on the basis of this analysis can support adaptive management
by introducing necessary changes to the project plan and to the supporting budget.
The decision-making process builds on the recommendations that have been developed, respond-
ing to significant changes identified during the monitoring process. This will result in a clear justifi-
cation for any proposed changes to the plan and budget.
It is important to clarify who will be involved in decisions about changes because:
= Representation of important stakeholders increases ownership and support for decisions.
= It will allow different views to be considered during the decision-making process.
= It can reduce delays by assigning certain decisions to staff more closely involved in project im-
plementation.
If an Operations Manual or Policy Document is developed at the start of the project, it can include
a section on decision-making responsibilities and processes. Such a document should be reviewed
occasionally to keep it up to date with any changes that have taken place in the organisation.
The manual or policy should identify the decisions to be made by:
» Staff.
» Management.
» The board or steering group of the project.
In consultation with:
» The target group, beneficiaries, donors and other stakeholders.
The principle of subsidiarity should be kept in mind when deciding how decisions will be made.
According to this principle, solutions to problems should be identified and acted on as close to the
source of the problem as possible.
If you act on the basis of this principle, each succeeding level of management will only deal with
the problems that cannot be resolved at a lower level. This can relieve management from a great
burden of having to approve all changes. It also means that management can focus on dealing with
blockages in the system that cannot be addressed at a lower level.
This principle also supports the view that the focus of M&E is on providing project staff with timely
information for implementing the plan. If all decisions have to be passed through the project’s man-
agement hierarchy, the staff may be blocked from developing appropriate and timely responses to
emerging problems.
Monitoring and evaluation: tools for biodiversity conservation and development projects
How will decisions be made?
Agreement on who will take various decisions will not, by itself, result in healthy decision-making
processes. It is also important to pay attention to the way the decisions will be made.
It is useful to keep in mind that organisational culture has an important influence on M&E.An or-
ganisational culture that is open to feedback, curious and respectful of others, will support timely
decision-making that includes the views of interested stakeholders. The participation of these
stakeholders can end up having little or no influence if a small group ultimately takes decisions in
an autocratic manner. There may be times when decisions have to be taken quickly, without much
time for consultation with affected stakeholders. There may be other occasions when legal or con-
tractual requirements direct the process, overriding the views of certain stakeholders.
There are a number of methods that a group can use to make decisions. Some of these options are
identified on the following pages.
What is being decided?
Decision-makers are being asked to approve a recommended change. They may discuss whether or
not they think that the recommendation is appropriate to the change that it is responding to. They
may want to look at the information that has been used to make the analysis. If major changes are
being recommended, the decision-makers may want to carry out a diagnostic study to be sure that
that this is an appropriate response.
Decision-makers may want to adapt the recommendation. It would be helpful to consult with the
people who made the analysis to be sure that they are not overlooking some important reasons
for the initial recommendations.
When the decision to approve a recommendation is made, it will be necessary to ensure that it is
properly embedded in the project by identifying:
e Whowill carry it out?
Assign responsibility for implementing any changes.
e By when should it be implemented?
Clarify how much time the implementers will have to make the changes.
e How will the changes be monitored?
Agree on the indicators that will be used to check whether or not the recommendation has
been implemented, and whether it is having the desired influence on the project and its context.
If staff are to implement the decisions in the ways intended, they need to understand the reasoning
behind the decisions. This is more likely to happen if staff are also involved in the process of col-
lecting and analysing information. No matter to what extent staff have been involved, communica-
tion still plays an important role in ensuring that people are clear about why decisions have been
made, and what is expected of them. Staff may be reluctant, for a range of reasons, to carry out the
decisions. It will be important to assess why this resistance has developed, so that an appropriate
response can be developed.
Monitoring and evaluation: tools for biodiversity conservation and development projects
Why resistance matters
Resistance is any force that slows or stops movement. People resist in response to something. This
could be in response to something we do, or to something we propose. Changes do not fail because of
resistance—they often fail because of the way in which we respond to resistance.
Three levels of resistance can be identified.
e Level |: Based on information
(‘| don’t understand it.’)
This level of resistance is based on the information, the facts, figures and ideas, the presentations
and the logical arguments that are often used to persuade people that change is necessary.A typical
response is to offer even more information when something completely different is called for.
e Level 2: Physiological and emotional reactions to the change
(‘I don’t like it.’)
This level of resistance is based on fear—people fear that they will lose face, friends, perhaps even
their jobs. This fear is physiological and can be triggered without conscious awareness. People ex-
periencing these symptoms have a diminished ability to listen, and conversations can become very
strained.
e Level 3: Bigger than the current change
(‘I don’t like you.’)
People are not resisting the change in itself—they are resisting the source of the change. They may
be resistant because of their relationship with the person or group perceived as responsible for the
change, and whom they are seen to represent. Level 3 resistance is also the area of cultural, religious
and racial differences. It is unlikely that people will place much value on the ideas and opinions of
the other side.
The challenge in working with resistance is to become aware of the level from which the resistance is
coming, and to deal with it through appropriate strategies. Level | tactics of communication will seldom
work at Level 2 and 3.
(Maurer, undated; Thaw 2002)
89
90 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Working with resistance
Having read the piece on ‘Why resistance matters’, make some notes on the following:
= Where have you encountered resistance?
= What was your response to this resistance?
* What do you think was the value of this resistance?
Se)
Monitoring and evaluation: tools for biodiversity conservation and development projects
w y
Different methods for decision-making in groups have been identified. The chosen method should be
appropriate in the context you face, taking account of such factors as the time available, the past his-
tory of the group, the kind of task on which you are working, and the kind of climate you are trying to
develop in the group.
e Decision by lack of response
Group members may offer ideas for the group to consider. If there is no response to the idea, the
group's lack of response is an effective decision not to support the idea.
e Decision by formal authority
The group may set up or inherit a power structure in which it is understood that someone in
authority will make the decisions. While group members can generate and discuss ideas, the
authority figure will make the decisions when she or he feels that they have heard enough
information as a base for the decision.
e Decision by self-authorisation or minority
A small number of group members may make suggestions that result in decisions without the
consent of the majority of the members. If there are no responses to the suggestions, the silence is
taken as support for the decision. This assumption may be wrong, especially if people feel reluctant
to be seen as the only voice opposing the suggestion or delaying the group from making a decision.
e Decision by majority rule
Members of the group can vote on a proposal or share their views, and if there is a majority view,
it is taken as a decision by the group. However, the decision may not have a high level of support if
some group members feel that there was insufficient time to discuss the ideas before voting took
place, or that the issue was too important to be decided on the basis of majority voting.
e Decision by consensus
Decisions by consensus occur when the group has had sufficient time to discuss the issue, with all
the members feeling that they have had a fair chance to influence the decision. Communication plays
an important role, as group members are more likely to support a decision with which they do not
fully agree if they feel that the group has listened to their concerns.
e Decision by unanimous consent
The decision is made with the full support of all the group members. While this may be necessary in
certain key decisions about the project, it will not always be appropriate. It can be a very inefficient
way to make decisions, as it could delay the group from reaching a conclusion.
(Schein 1999)
92 Monitoring and evaluation: tools for biodiversity conservation and development projects
In Summary:
—
REFERENCES AND FURTHER READING
MAURER, R. Undated.Why resistance matters. http://www.beyondresistance.com/resistance_to_change.htm (accessed
January 2008).
SCHEIN, E.H. 1999. Process consultation revisited: building the helping relationship. Addison-Wesley, Reading MA.
THAWY, D. 1996-2002. Ideas for a Change Series. Olive Publications, Durban.
Monitoring and evaluation: tools for biodiversity conservation and development projects 93
Section 6: SHARING THE LEARNING (REPORTING)
While the notion of reporting is most commonly associated with donor requirements, most
projects have a range of stakeholders who want to be kept informed and to whom the project is
accountable. Keeping them up-to-date is important so that they can support critical decisions that
the project might have to make in the future.
Project-level reporting serves the needs of a number of stakeholders by:
= Generating the inputs required for programme-level monitoring.
= Meeting the very specific requirements of small grant-funders.
* Acting as a vehicle for accountability to the community in which the project operates, both
internal (staff) and external (beneficiaries, target group, interested and affected parties).
= Acting as a vehicle for communicating the learning to the project’s partners and peers.
The reporting needs of each of these stakeholders are likely to differ in respect of both format and
content. Keeping them all informed could prove to be time-consuming. Our aim here is to provide
you with tools to develop effective and efficient reporting systems that enable you to disseminate
the information and learning assembled through the M&E process to all stakeholders, without du-
plication or undue effort.
By the end of this section, you will have:
# Explored the notion of accountability and the range of stakeholders to whom project reporting is
directed.
= A better understanding of what it is that the various stakeholders want to know.
= A better understanding of donor reporting terminology.
= Understood how sharing the learning can contribute to improvement in your own practice.
Developed a framework for more efficient reporting within the context of your project.
94 Monitoring and evaluation: tools for biodiversity conservation and development projects
6.1 WHAT IS ACCOUNTABILITY?
Accountability is about building and maintaining relationships based on learning and mutual respect.
Interacting with your target group, project partners and beneficiaries as you exchange ideas, ex-
periences and offer support, provides many opportunities for practising accountability. By this we
mean accounting to all stakeholders for what the project committed itself to achieve and for creat-
ing opportunities for the sharing of learning during the life of the project.
Accountability is also about acknowledging the support the project has received from various
stakeholders. This can take place in different ways, but the purpose is to report to the stakeholders
by explaining what has been accomplished with the resources that they made available to support
the project. Their support may have been in the form of people, finances, equipment and supplies,
or it may have been political (lending their support and influence).
What do the stakeholders want to know?
As they are involved in the project in different ways, the interests of stakeholders will differ. Meth-
ods for distinguishing between these interests during the process of developing your M&E frame-
work are discussed in Section |.In this discussion we explore the specific interests of six different
groups of stakeholders.
e Beneficiaries
Beneficiaries are the people who ultimately benefit from the project intervention, for example
the people of the Cape Floristic Region. In some cases, particularly in the biodiversity conser-
vation sector, the benefits accrue to people indirectly because project outcomes are directed
at measures that safeguard the environment and resources on which people depend. In such
instances, beneficiary interests can be served through the same reporting channels as for the
public.
In other instances, where there are people who are direct beneficiaries of project interventions,
they are likely to take a closer interest in the project. They should be kept informed on what the
project has accomplished, and would be represented in the group of interested and affected par-
ties, possibly along with members of the target group.
e Target group
They are the people who are targeted by the project team as a vehicle for achieving the project
purpose. It is possible that a representative group or committee of the project’s target group
was formed during the project establishment phase and this is the group to whom the project
can report.
The purpose of communicating with the target group is to:
* Report on what progress the project is making.
= Inform them of any changes in the products, goods and services that the project is delivering.
= Receive their feedback on how they perceive the quality and usefulness of your support.
Information received from this group is frequently used to assess the progress and achievements
of the project. They are thus a key stakeholder in the process of accountability. In evaluation
processes, they are often asked for much information and given relatively little back in return.
Monitoring and evaluation: tools for biodiversity conservation and development projects 95
While it is common to send written reports to donors, reporting to the target group can best
be served through face-to-face interactions. This creates opportunities for you to be creative in
the way you report to them. The most appropriate and efficient forms of reporting are likely to
be:
« Visual presentations of the project’s experiences and lessons, using video or electronic pro-
jection.
= Charts, diagrams and other graphic media.
In preparing the material, you should take into account the venue where the presentation will
take place and whether the representatives will, in turn, be reporting back to other groups.
Annual audits are carried out with the farmers who are participating in the programme.There could be more
communication with them, on an ongoing basis, about their needs, and perceptions of the benefits of the
programme. There is this assumption that people will be happy if they have signed up. The management plan is
based on their original list of needs. It is likely that they are still the same, but maybe we are not tapping into
landowner needs. Our resources do not stretch that far.
(Conservation Partnerships Programme)
e The public
The general public can be an important audience for project accountability. Rather than re-
porting simply on what has happened, this form of accountability can take place with a view
to influencing the public. They can be influenced to do things that may be to the benefit of the
project and its objectives. The public can be an important source of contributions, in the form
of donations and volunteers. Changes in behaviour and attitudes can help shape public percep-
tions about the issues that the project is addressing, thereby enhancing its broader impact.
Members of the public who are policy-makers, even if they are not connected to the project in
any way, are a potentially significant audience for project reporting. They include elected officials,
such as members of parliament at national and regional levels and city councillors, as well as ap-
pointed officials of government departments and agencies. By bringing the project’s experiences
and lessons to their attention, they can be influenced to make changes in policy, legislation and
operational requirements that will benefit a much larger population than the project could
reach directly.
This group is likely to be interested in:
= The changes that have occurred as a result of the project's intervention.
= The impact of the project on the lives of beneficiaries and the target group.
« The potential replicability of the project.
The most appropriate and efficient forms of reporting to this group are likely to be:
# Visual presentations at public meetings, using video or electronic projection.
« Feature articles, reports or letters to the editor in newspapers or magazines.
= A website or page on the partner programme's website.
96
Monitoring and evaluation: tools for biodiversity conservation and development projects
Collegial interest groups and peers
Reporting to collegial interest groups and peers results in sharing the lessons with others, and
building support for changes in practice. Dissemination of learning might take place at different
levels, depending on who was involved in identifying the lessons. Besides the sharing that oc-
curs within the project team (discussed in Section 2), reporting enables learning in the following
contexts.
* Across an organisation: The organisation may host a number of projects that can benefit
from each other’s experience. By paying attention to sharing lessons within the organisation,
there is a greater likelihood that the whole organisation can benefit from the experience of
one project.
= Between organisations: Organisations that are doing similar work can benefit from one
another’s experience. This can prevent organisations that work in the same sector from
repeating mistakes already made elsewhere.
This group is likely to be interested in:
» The project’s findings on the reasons for any divergence between targets and achievements,
and the strategy for adaptation.
» Lessons relating to methodology and management systems.
» Tools that proved to be useful to the project.
» The potential replicability of the project.
Care has to be taken with the way you disseminate lessons. There is a risk that you might lose
credibility in the eyes of your colleagues and partners if they think you are trying to impose
lessons on them. If the lessons are to be accepted and adopted by others, it is better if they
identify the relevance of lessons to their own context. This allows them to translate a general
lesson to the specific situation they are facing.
Furthermore, the nature of conservation work may mean that the lessons include scientific in-
formation, survey results and other forms of technical data. It is important to think about ways
to integrate this information into the project findings (and project reports) so that it can be
disseminated and shared with others.
Finally, the process of disseminating learning can serve as an opportunity for team learning. By
bringing people together to review their experience and the emerging lessons, the whole group
has an opportunity to link the learning to innovations in upcoming events and in the project
plans.
We have annual interprovincial get-togethers, 3-day peer learning bosberaads.We use that to try to integrate
other programme learnings.
(Conservation Partnerships Programme)
Monitoring and evaluation: tools for biodiversity conservation and development projects
e Host programme
Most projects are embedded within a host programme. Their relationship is through their com-
mon objectives: the outputs of the project contribute to the achievement of the programme’s
objectives. The host programme therefore needs information on the project’s achievements to
maintain its own M&E processes.
The host programme is likely to want written reports on:
* Actual achievements relative to planned targets.
97
» Any proposed changes to the logframe and an explanation of the reasons for the divergence
between targets and achievements.
The host programme may play a supportive role by facilitating the sharing of learning that
occurs within its component projects, and offering assistance to projects that may be having
difficulty in complying with donor reporting, in which case it would have an interest in all the
reporting outputs discussed here.
Donors
It is useful to remember that donors provide financial support because your project is in line
with their own objectives. They want regular written reports that describe project progress, its
context and financial history. These reports help them to assess whether resources are being
used in line with the plans, and what progress is taking place. They, in turn, often use this infor-
mation in their own reporting to their donors and supporters.
The information that they are interested in revolves around these questions:
# Is the project achieving its planned outcomes?
» Are the resources they contributed being used in an efficient and effective manner?
" Also, if the project is contemplating any major changes to the planned outputs or use of
resources, what are these planned changes, and what are the reasons and motivations for
making the changes?
Some ways to disseminate learning
e Communities of practice
These forums bring together people interested in learning from one another about a common
area of practice. Such groups can be formal or informal, operating with or without the support of
the organisation. Some communities of practice take place in person, as people share a common
workplace. An increasing number of such communities are making use of the internet by creating
electronic or virtual communities that permit them to interact with people doing similar work in
other parts of the country and around the world.
e Websites
These are a more recent forum for disseminating project lessons. If the project or organisation
has its own website on the internet, you can develop a section for ‘Emerging Lessons’. This can be
an additional way of disseminating any writing you have done. You can also make contributions to
websites maintained by other organisations.
98 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Newsletters, magazines and other media
Writing about your experience is another way to disseminate your learning. By writing and
publishing articles in the newsletters, magazines, blogs and other media read by your colleagues, you
have the opportunity to share your perspective on how your fellow professionals can improve their
practice. .
e Seminars and conferences
r
1
'
'
'
'
'
'
'
'
1
1
1
1
1 You can use the opportunities provided by seminars and conferences arranged by networks and
NGO coalitions as a further avenue to share your learning. You can prepare a presentation for
' such an event, or be part of a panel discussion in which you describe what you have been learning
! through your project experience.
1
!
1
1
'
1
1
1
1
1
1
1
1
1
1
1
‘
&
e Knowledge management
This is an organised way for an organisation to systematically capture and disseminate the
knowledge that is created within the organisation. It can make it easier for staff to locate people and
knowledge as they need it to help them solve problems they face in their work.A growing number
of NGOs are now paying attention to knowledge management programmes, especially for sharing
knowledge between projects and offices located in different geographic areas, and for informing
the support and policy work of a central office. Knowledge management relies on a strong culture
among people in the organisation for documenting and sharing what they know with one another.
(Collison & Purcell 2004; Bellanet website)
Monitoring and evaluation: tools for biodiversity conservation and development projects 99
6.2 REPORTING FORMATS
Many donors have developed specific formats for written reports, and funding contracts often
stipulate dates when reports should be submitted. It can mean much additional work for project
staff if the project’s own formats for gathering information and internal reporting differ from those
of the donor.
This becomes even more complicated if the project, or its host organisation, has multiple donors,
each with its own reporting formats and timeframes. Many donors are sympathetic to the re-
porting burden faced by project staff. ln some cases it may be possible to develop a common and
agreed format for reporting that meets the accountability needs of a number of donors. For ‘small
grants’ beneficiaries, reporting can be framed as qualitative commentary because donors’ specifica-
tions are fairly loose.
Some examples of the reporting information required by small grants donors include:
= Updates on progress made against objectives.
= Evaluation of accomplishment/ successes.
* Asummary describing the project’s conservation impacts.
= Progress towards the achievement of the objectives.
« Unexpected impacts.
# Intended or unintended negative consequences.
« Disappointments or failures during implementation.
It can be difficult for people who are not experienced in reporting to respond to open-ended
specifications such as these, and as a result they may fail to provide the information that donors
are looking for.
A framework for progress reports
L
R
While many donors have developed formats for written reports, not all donors require their grantees
to use a specific format. If you have flexibility in the way you report, or if you are trying to negotiate
with donors for a format that is more appropriate to your needs, you might consider including the fol-
lowing elements:
= Current status of the project.
= Major activities undertaken during the period under review (tasks completed or in progress within
each activity).
# The project’s financial performance.
= Estimates of the number of beneficiaries serviced, relative to targets, and their responses.
=" Current and potential problems (this includes a description of the problems, and remedial actions
planned or recommended).
= Project plans and schedules for major activities during the next review period.
This framework can assist your communication with donors and other stakeholders, and reveals the
reasons why you are considering changes. The reports can themselves be used in future monitoring, as
they will serve as a summary of what you intended to accomplish during the next period.
1
1
1
1
1
1
1
1
1
1
1
1
1
1
!
i
1
1
1
1
1
1
1
1
1
1
1
j
1
1
; (Casley & Kumar |987)
1
100 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Using the action learning cycle for report writing
Review a recent monitoring and evaluation report produced by your project.
Assess what percentage of the report fits with each of the four parts of the action learning cycle:
Experience.
Reflection.
Insight and understanding.
Planning.
Are you satisfied with this distribution? Are any of the components absent or underrepresented?
How might you change the way you prepare and produce reports so that they include a better distribu-
tion of the various elements of the action learning cycle?
Monitoring and evaluation: tools for biodiversity conservation and development projects 101
Unpacking reporting terminology
The most commonly used terms in donors’ reporting specifications are:
e Progress relative to objectives
This may be phrased as ‘progress made against objectives’ or ‘progress towards the achievement
of the objectives’. You will find a description of the project’s objectives in the logframe or contract
with the donor. The objectives are likely to be referred to as ‘results’ or ‘outputs’. In your report,
you have to list each of these as a heading and below that objectively describe all the activities that
have been undertaken towards achieving that result or output.
e Evaluation of accomplishments or successes
Evaluation requires a comparison of what has been achieved against what was planned. The project
plan is a useful source of reference for what was planned.Where there is a difference between the
two, the evaluation should provide some insights as to why, and what can be learned from it. (Refer
to Section 4.)
Below are some terms used by Small Grants Funders. Given their open-endedness, it is useful to discuss
with your donors their understanding of the phrases and come to an agreement on the way you will
use them in your reports.
e Conservation impacts
While ‘impact’ is sometimes used to refer to a long-term goal, in the context of project-reporting
it is an enquiry into the immediate impact that the project is having in its local context. What is the
project contributing immediately towards the long-term goal of biodiversity conservation?
e Unexpected impacts
Sometimes projects have positive outcomes that were not envisaged or intended in the project plan.
These ‘side-effects’ or ‘by-products’ may represent important benefits and can be extremely useful
learning opportunities.
e Intended or unintended negative consequences
Just as a project may generate unexpected benefits, so too can it result in unexpected negative
consequences. In some cases, negative consequences may have been anticipated as intended and
unavoidable outcomes of a project intervention.
e Disappointments or failures during implementation
This category of information provides the opportunity for presenting information about the ab-
sence of enabling factors that may have impacted on the project’s progress or success.
The last three of the above terms represent opportunities for the reporter to explore the ‘why’ of any
divergence between planned and actual outputs and lay the foundation for motivating for a change to
the project plan.
102 Monitoring and evaluation: tools for biodiversity conservation and development projects
By analysing elements of the previous discussion, we can see that there is some overlap in the
interests of the different stakeholders:
Public/
bene- Host pro-
Reporting content ficiaries gramme
Progress relative to objectives v v v
Localised impact of project outputs v v
Potential replicability of the project v v v
Lessons relating to methodology and y yy ©
management systems
Tools that were useful to the project v v
Efficiency and effectiveness of resource ie V
use
Reasons for divergence between targets
and achievements, and strategies for v v v
adaptation
The information needed by the host programme is a subset of the information needed by the do-
nors, both of which are written reports.
Likewise, in preparing presentations for the general public, the content would be the same as for
beneficiaries, with some of the more detailed information on localised and direct benefits and
impacts omitted.
The sharing of information with collegial interest groups and peers is likely to be focused on learn-
ing, and presented in the context of a seminar, conference or similar collegial gathering.As with any
audience, it is important that the material is tailored to the needs and interests of this particular
group, even though there may be overlap with material used for beneficiaries, for example. Herein
lies its usefulness: this may be the only opportunity that project staff have for the kind of objectivity
and reflection that is needed to derive lessons from project practice. By encouraging staff to partic-
ipate in these events, and to prepare papers and articles that disseminate the learning, the organisa-
tion benefits, not only from the exposure, but also from the reflection that is enabled through the
process of preparing the material.
Whether reporting content and format is externally imposed or driven by communications-based
criteria,a common set of normative principles apply to project reporting at all levels, inter alia:
= Reporting should draw directly from information gathered through M&E activities. Project de-
velopers should not be required to report on aspects that have not been monitored. Reporting
requirements thus have to become an informing aspect in the design of the M&E framework.
Monitoring and evaluation: tools for biodiversity conservation and development projects
Over-reporting leads to reporting fatigue. Reporting formats should have clearly stated objec-
tives and include only fields that will actually be useful to the users of the information. The
frequency and timing of reporting should coincide with related activities in the project cycle.
Duplication in reporting (to the various constituencies) should be avoided.
The challenge of communicating with different audiences is to promote a dialogue between the organi-
sation and its stakeholders, and to prevent reporting from being a one-way flow of information.
e Bridging communication gaps
Differences in the status and background of stakeholders, which complicate the communication
process, can be addressed with time, and with appropriate attitudes and behaviour.
e Aggregating lessons
The tension between presenting broad conclusions that feed strategy and policy work, and the rich-
ness that lies in the detail of local situations.
e Reaching out
Drawing the attention of specific groups to new lessons in credible and relevant ways.
e Communication in reverse
Feeding back to informants the decisions taken as a result of their contributions to the monitoring
process.
(Barnard & Cameron 2000)
103
104 Monitoring and evaluation: tools for biodiversity conservation and development projects
Exercise: Building a matrix of your project's reporting requirements
By analysing the requirements of each of the stakeholders to whom your project is accountable, you will
be able to identify the commonalities that will enable you to streamline your reporting system.
PROJECT STAKEHOLDERS:
AREAS OF INTEREST:
Progress relative to objectives
Localised impact of project outputs
Potential replicability of the project
Lessons relating to methodology and management systems
Tools that were useful to the project
Efficiency and effectiveness of resource use
Reasons for divergence between targets and achievements,
and proposals for adaptation
PRESENTATION FORMAT:
WRITTEN FORMAT:
Commonalities:
Monitoring and evaluation: tools for biodiversity conservation and development projects 105
In} Summary,
In Section 6 we:
= Examined the accountability interests of the various stakeholders and identified what this
means for project reporting.
= Identified the common areas and hierarchy of information needed for reporting purposes.
= Looked in more detail at:
» Ways of disseminating learning across the organisation and between organisations.
» A generic framework for reporting on progress.
» The terminology commonly used in donor reporting.
» Methods for communicating with your key audience.
= Provided a matrix form to help you to analyse your project’s reporting requirements.
By interacting with stakeholders and professional colleagues, you create the opportunity for re-
flection and learning in your own practice and for your experience to influence the thinking and
actions in other organisations. Their experience of introducing your ideas will stimulate their
own reflection and learning, resulting in ongoing improvements in their practice and impact.
REFERENCES AND FURTHER READING
BARNARD, G. & CAMERON, C. 2000. Efela Synthesis Report: evaluation feedback for effective learning and accountability.
Institute of Development Studies, Brighton. http://www.oecd.org/dataoecd/ | 0/29/2667326.pdf (accessed January
2008).
BELLANET Undated. Knowledge management for development organisations. http://home.bellanet.org (accessed
January 2008).
CASLEY, D.J. & KUMAR, K. 1987. Project monitoring and evaluation in agriculture. Johns Hopkins University Press,
Baltimore.
COLLISON, C. & PARCELL, G. 2004. Learning to fly: practical lessons from one of the world’s leading knowledge companies.
Capstone, Oxford.
Monitoring and evaluation: tools for biodiversity conservation and development projects 107
This handbook has introduced a number of ideas and resources that can help you as you prepare
to assess your work by engaging in a monitoring and evaluation process. We have also offered ma-
terial to assist you in becoming more mindful about your practice of project M&E.
There is no magic formula that you can follow to develop an effective project monitoring system
and practice. We have introduced a broad framework and outlined the elements of an approach
to your M&E practice that can be adapted to fit the needs and context of your project and your
organisation.
As you explore this issue further, remember that M&E takes place to support the implementation
of your project. It does so by allowing you to step back from your work occasionally to consider
what you have achieved, and the changing context in which you are situated. M&E exists to serve
the project—it should not develop a life of its own, independent of the information and reflection
needs of the project.
Ultimately, M&E can provide you with insight and perspective. It can help you to find things in your
work to value, and to identify practices to change. By reflecting on your work and your achieve-
ments, you will be equipped with resources to begin a new cycle of analysis and planning, devel-
oping new projects to support people in conserving biodiversity and meeting their development
needs.
We hope that you have found this component of the SANBI Biodiversity Series helpful in your prac-
tice and work. The ongoing cycle of planning, implementation and reflection allows us to be con-
tinual learners, both as individuals and as organisations.And this can only be to the benefit of the
people with whom we work and the environments in which we work.
Monitoring and evaluation: tools for biodiversity conservation and development projects 109
FURTHER READING
There is an extensive virtual library that can provide more information, resources, approaches and
tools that complement this handbook. The material presented in the handbook draws on two main
traditions:
e Participatory Learning and Action (PLA)
The PLA tradition builds on the experience of participatory tools such as Participatory Rural
Appraisal (PRA). It promotes the use of methods in a participatory manner, allowing respond-
ents to be involved in deciding which methods are appropriate, designing them, and collecting
information. By involving participants in the analysis of information for significant changes, there
is a greater likelihood that project stakeholders, as well as project staff, will learn from the ex-
perience.
A related practice has been developing among development practitioners in many countries in
recent years. It is known as Participatory Monitoring and Evaluation (PME).
e Social Sciences
Some development practitioners and researchers are interested in people, groups and organisa-
tions, and the relationships between them and their surrounding environment. They have de-
veloped tools to allow them to investigate what has taken place, and what people think about
these things. While the tools can collect quantitative or qualitative information, many practition-
ers are more interested in qualitative information.
These two traditions are built on a strong conceptual foundation and have been improved through
many years of field-based application by practitioners all over the world. However, some adaptation
may be needed to ensure that they fit your specific context.
A number of toolkits and books that provide additional detail on various methods for project
monitoring and evaluation have been written in recent years. They provide more extensive de-
scriptions and examples than what is possible in the space available in this handbook. Readers are
encouraged to make use of these resources for more explanation on the methods outlined on the
previous pages.
# DflD 2002.Tools for development. London, UK.
This toolkit contains a helpful overview of approaches and techniques that can be useful for ef-
fective project implementation. Available at www.dfid.gov.uk/pubs/files/toolsfordevelopment.pdf.
= FEUERSTEIN, M.T. 1986. Partners in evaluation: evaluating development and community pro-
grammes with participants. Macmillan, London.
This book provides a wide range of methods for participatory monitoring and evaluation. Writ-
ten in an easy-to-understand format, with clear illustrations.A subsidised edition of the book is
available at low cost from www.talcuk.org.
= GUIT, |. 1998. Participatory monitoring and impact assessment of sustainable agriculture ini-
tiatives. SARL Discussion Paper No. |. International Institute for Environment and Development
(IIED), London.
This handbook describes the author’s experience in developing key elements of an approach to
project monitoring. It details 20 methods, and is available from www.iied.org.
110
Monitoring and evaluation: tools for biodiversity conservation and development projects
IUCN. PME course documents.
Available at http://www.iucn.org/themes/eval/documents2/pme_concepts/pme_short_train-
ing course_OlI.pdf.
IUCN. Evaluation handbook (on managing an external evaluation).
Available at http://www.iucn.org/themes/eval/documents2/eval_handbook/handbook_eng.pdf.
Material made available by the Network of Networks on Impact Evaluation (NONIE).
http://www.worldbank.org/ieg/nonie/docs/IE_statement_draft.doc.
http://www.worldbank.org/ieg/nonie/docs/guidelines subgroup 2.doc.
ODI 2005. Successful communication. London, UK.
The toolkit lists a variety of practical approaches that can be used for communicating a
project’s purpose and messages to key stakeholders and beneficiaries. Available at http://www.
odi.org.uk/RAPID/Publications/Documents/Comms_toolkit.pdf.
ODI 2006. Tools for Knowledge and Learning. London, UK
The booklet offers a range of tools that can be used by a project team throughout the project
implementation process to help with internal management and quality improvements. Available
at http://www.odi.org.uk/RAPID/Publications/Documents/KM_toolkit_web.pdf.
PRETTY, J.N., GUIT, l., THOMPSON, J. & SCOONES, |. 1995. Participatory learning and action: a
trainer’s guide. International Institute for Environment and Development (IIED), London.
An introduction to PLA, the guide also includes 100 exercises that can be used in working with
groups, gathering and analysing information, and for decision-making. Available from www.iied.
org.
WADSWORTH, Y. 1997. Do it yourself social research, edn 2.Allen & Unwin, St Leonards.
This is the second edition of a popular Australian manual, full of practical advice on how to use
many of the common social science research tools, such as interviews and surveys. Available
from www.allen-unwin.co.au.
Monitoring and evaluation: tools for biodiversity conservation and development projects It
BIBLIOGRAPHY
ABBOTT, J. & GUIJT, |. 1998. Changing views on change: participatory approaches to monitoring the environment. SARL
Discussion Paper No. 2. International Institute for Environment and Development, London. http://www.iied.org/
pubs/pdfs/6 | 40IIED.pdf (accessed January 2008).
ACTION RESEARCH. http://www.scu.edu.au/schools/gcm/ar/arp/arphome.html (accessed January 2008).
BARNARD, G. & CAMERON, C. 2000. Efela synthesis report: evaluation feedback for effective learning and
accountability. Institute of Development Studies, Brighton. http://www.oecd.org/dataoecd/ | 0/29/2667326.pdf
(accessed January 2008).
BELLANET. Knowledge management for development organisations. http://home.bellanet.org (accessed January 2008).
CASLEY, DJ. & KUMAR, K. 1987. Project monitoring and evaluation in agriculture. Johns Hopkins University Press,
Baltimore.
CAVILL, S. & SOHAIL, M. 2007. Increasing strategic accountability: a framework for international NGOs. Development in
Practice 17,2.
CDRA 2001. Measuring development: holding infinity: a writing from the CDRA Annual Report 2000/2001. http://www.
cdra.org.za (accessed January 2008).
CIDA 1997. Guide to gender-sensitive indicators. http://www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/Policy/$file/
WID-GUID-E.pdf (accessed January 2008).
COLLINGWOOD, C. 2007. Revealing practice, re-imagining purpose, claiming our place Biennial Practice Conference,
McGregor, May 2007. http://www.biennial.cdra.org.za (accessed January 2008).
COLLISON, C. & PARCELL, G. 2004. Learning to fly: practical lessons from one of the world’s leading knowledge companies.
Capstone, Oxford.
DAVIES, R. 1997. Lesson learning: how will we recognise it when we bump into it? http://www.mande.co.uk/docs/
lessons.htm (accessed January 2008).
DE RUIJTER,A., DIETZ, T..VAN GONGEN, E., HELMSING, B. & KNORRINGA, P. 2006. Evaluation of the Theme-based
Co-financing Programme. Cross-cutting study: monitoring and evaluation. Prepared for Ministry of Foreign Affairs
of the Netherlands. http://www.tmf-evaluatie.nl/eindrapportagem20M&E.doc (accessed January 2008).
ELDIS. Participatory monitoring and evaluation. http://www.eldis.org (accessed January 2008).
ESTRELLA, M. (ed.) 2000. Learning from change: issues and experiences in participatory monitoring and evaluation.
International Development Research Centre, Ottawa.
ESTRELLA, M. & GAVENTA, J. 1998.Who counts reality? Participatory monitoring and evaluation: a literature review.
IDS Working Paper 70. Institute of Development Studies, Brighton. http://www.ids.ac.uk/ids/bookshop/wp/wp/70.pdf
(accessed January 2008).
FEUERSTEIN, M.-T. 1986. Partners in evaluation: evaluating development and community programmes with participants.
Macmillan, London.
FOWLER, A. 2000. The virtuous spiral: a guide to sustainability for NGOs in international development. Earthscan, London.
GOOLD, L. 2006. Working with barriers to organisational learning. Bond, London. http://www.bond.org.uk/pubs/briefs/
olbarriers.pdf (accessed January 2008).
GOSLING, L. & EDWARDS, M. 1995.Toolkits: a practical guide to assessment, monitoring, review and evaluation.
Development Manual 5. Save the Children, London.
GUIJJT, |. 1998. Participatory monitoring and impact assessment of sustainable agriculture initiatives: an introduction to
key elements. SARL Discussion Paper No. |. International Institute for Environment and Development, London.
http://www.iied.org/pubs/pdfs/6 | 39IIED.pdf (accessed January 2008).
HARRISON, R. 1995. The collected papers of Roger Harrison. McGraw-Hill, New York.
IDS 1998. Participatory monitoring and evaluation: learning from change. IDS Policy Briefing | 2. http://www.ids.ac.uk/
UserFiles/File/publications/policy_briefs/PB|2.pdf (accessed January 2008).
INTERNATIONAL INSTITUTE FOR ENVIRONMENT AND DEVELOPMENT. Participatory resources. http://www.iied.
org (accessed January 2008).
KELLEHER, D., McLAREN, K. & BISSON, R. 1996. Grabbing the tiger by the tail: NGOs learning for organizational change.
Canadian Council for International Co-operation, Ottawa.
112 Monitoring and evaluation: tools for biodiversity conservation and development projects
KNOWLES, M. 1980. Cited on the Quay Communications website. www.quaycom.com/library_images/white_papers/
adult_learning.doc (accessed January 2008).
LAWRENCE,A. (ed.) 2002. Participatory assessment, monitoring and evaluation of biodiversity. ln Summary of the
ETFRN internet discussion 7—25 January 2002. http://www.etfrn.org/etfrn/workshop/biodiversity/documents/
dissum.doc (accessed January 2008).
LIEB, S. 1991. http://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/ teachtip/adults-2.htm (accessed
January 2008).
MONITORING AND EVALUATION. M and E News. http://www.mande.co.uk (accessed January 2008).
MAURER, R. No date.Why resistance matters. http://www.beyondresistance.com/resistance_to_change.htm (accessed
January 2008).
PRETTY, J., GUIJT, |, THOMPSON, J. & SCOONES, |. 1995. Participatory learning and action: a trainer's guide. International
Institute for Environment and Development, London.
RAELIN, J. 2000. Work-based learning. Prentice Hall, Upper Saddle River, NJ.
RANDEL, M. 2002. Planning for monitoring and evaluation. Olive Publications, Durban.
REELER, D. 2001. Unlearning—facing up to the real challenge of learning. http://www.cdra.org.za (accessed January
2008).
ROGERS, PJ. & WILLIAMS, B. 2006. Evaluation for practice improvement and organisational learning. In |. Shaw, J. Greene
& M. Mark (eds), Handbook of evaluation. Sage Publications, London.
SCHEIN, E.H. 1999. Process consultation revisited: building the helping relationship. Addison-Wesley, Reading MA.
SENGE, P. 1994. The fifth discipline fieldbook. Nicholas Brealey, London.
SENGE, P. 1999. The dance of change: the challenges to sustaining momentum in learning organisations. Currency/Doubleday,
New York.
SHAPIRO, J. 1995. Financial management for self-reliance: a manual on managing the finances of a non-profit organisation.
Olive Publications, Durban.
SHAPIRO, J. 1996. Evaluation: judgment day or management tool? Olive Publications, Durban.
SMILLIE, |. & HAILEY, J. 2001. Managing for change: leadership, strategy and management in Asian NGOs. Earthscan, London.
SOAL, S. 2001. How do we know what difference we are making? Reflections on measuring development in South
Africa.Adapted from a presentation to a workshop between Christian Aid and South African partners, Durban,
October 2001. http://www.cdra.org.za (accessed January 2008).
SOAL, S., HILL, S. & REELER, D. 2007. Evaluation: a developmental approach. http://www.cdra.org.za (accessed January
2008).
STAKE, R.E. 1995. The art of case study research. Sage Publications, California.
STEM, C., MARGOLUIS, R., SALAFSKY, N. & BROWN, M. 2005. Monitoring and evaluation in conservation: a review of
trends and approaches in conservation biology. Blackwell, Gainesville Florida.
TAYLOR, J. 1998. NGOs as learning organisations. http://www.cdra.org.za (accessed January 2008).
TAYLOR, J., MARAIS, D. & KAPLAN, A. 1997. Action learning for development: use your experience to improve your
effectiveness. Juta, Cape Town.
THAW, D. 1996-2002. Ideas for a Change Series. Olive Publications, Durban.
UPHOFF, N. & COMBS, J. 2001. Some things can’t be true but are: rice, rickets and what else? http://www.eldis.org/
fulltext/TRUE-RRR.pdf (accessed January 2008).
WADSWORTH, Y. 1997. Do it yourself social research, edn 2.Allen & Unwin, St Leonards.
WADSWORTH, Y. 1997. Everyday evaluation on the run, edn 2.Allen & Unwin, St Leonards.
WOODHILL, J. 2006. Monitoring and evaluation as learning: rethinking the dominant paradigm. In J. de Graaff, J.
Cameron, S. Sombatpanit, C. Pieri & J. Woodhill (eds), Monitoring and evaluation of soil conservation and watershed
development projects. World Association of Soil and Water Conservation. Science Publishers, New Hampshire, USA.
WWE (World Wide Fund for Nature) 2005. Basic guidance for step 2.2: monitoring plan. http://assets.panda.org/
downloads/2_2_monitoring_plan_01_11_O5.pdf (accessed January 2008).
YIN, R.K. 2003. Case study research: design and methods. Sage Publications, California.
Monitoring and evaluation: tools for biodiversity conservation and development projects 113
Appendix 1: GUIDELINES FOR TRAINERS
INTRODUCTION
In this guide we suggest ways of using it for the purpose of training and building the capacity of
others in M&E. In particular, it is aimed at project managers who are interested in building the M&E
capacity of project staff. It is also intended for project developers who are helping to support new
project managers in developing their M&E skills and understanding.
The guidelines are presented as exercises and process tips for dealing with key topics in each sec-
tion of the handbook.
The first, self-reflective introductory exercise is intended to immerse project managers and staff
in their own experience of M&E, before they start to develop a plan for the project. It is also an
exercise that project developers could use for themselves.
Exercise
Think back over your personal and working life, trying to identify all experiences of monitoring and
evaluating. Consider the following questions:
When have you been an active participant in M&E? When have you been a recipient?
What were these significant events, e.g. exams, driving test, performance appraisal?
What were the roles you played?
Classify the evaluation into types, e.g. internal, external or self-evaluation.
What were the contexts of each evaluation?
Map this information out in whatever way works best for you, e.g.a timeline or a mind map.
Make notes on these questions:
= = What have | learnt about M&E during my life?
= What claims can | make about M&E (things | know to be true)?
= What questions do | have?
(Source: Project Developers’ Forum, |8—19 March 2008)
114 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 1: DEVELOPING AN Mae PLAN
There are a number of different themes that can be explored in this section. We have identified
four key themes.
e Theme |—Working with questions as a basis for developing a plan
Here we focus on developing an M&E plan, using questions to guide the formulation of the plan.
The questions encourage thinking and in-depth inquiry about what the project is generally try-
ing to achieve, as well as the specific components of M&E. For example, who will co-ordinate
M&E? How? When? and so on.
It is likely that even if someone is relatively new to M&E, they will have some experience to
draw on. Furthermore, for project managers who are formally adopting an M&E system for the
first time, there are likely to be existing practices in the project that can feed directly into this
framework.
Therefore a first step in supporting someone who is developing an M&E plan is to help them to
identify what is already in place, and what is currently happening.
Exercise
Introduce the eight key questions identified in the handbook Section |: Developing an M&E plan,
page 12.
Provide a brief explanation of what each question is asking for. In some cases, you might have to
give reminders about terminology. For example, when looking at the objectives of the project, you
could use this as an opportunity to clarify the ‘hierarchy’ of objectives that are used in the Logical
Framework Approach (LFA). For question 2 (Who will use the information collected?), you could
help them understand the different categories of end-users by naming them, i.e. beneficiaries,
target group, project staff, management and donors. For question 3 (How are indicators used and
developed?), you might check that participants understand the difference between quantitative and
qualitative indicators, and so on.
Ask the individual or group to spend one hour working on their own, or in pairs, making brief notes
on what is currently in place in their project regarding the eight questions.
Debrief the exercise by asking:
* How well do we understand our project? Are there any areas where we would benefit from
knowing more about our project? (‘Reviewing your objectives’ on page 16, could also be used
to check understanding of the project.)
What are the strengths of our existing M&E activities?
Where are there gaps?
What can we do to address these gaps?
Who can help us with this?
By when?
e Theme 2—Developing indicators
Developing indicators requires rigour and creativity. Rigour ensures that indicators are precise
and measurable, and creativity is needed to identify indicators that can be measured in the
required time, with the available resources and capacity.
Monitoring and evaluation: tools for biodiversity conservation and development projects 115
Before you begin, check that there is shared understanding of LFA terms within the project. (It
is likely that project staff will have some familiarity with the language.) Draw out their under-
standing of the key terms: objectives, outcomes, outputs and indicators.
From this basis, it is possible to develop useful project indicators.
Exercise
|. Explore the different aspects of quantitative and qualitative indicators, generating examples of each.
2. Identify the level of project objectives for which you want to develop indicators.
3. Follow the guidelines on ‘How do you develop indicators?’ on page 24.
e Theme 3—Unpacking the notion of ‘participation’
This theme helps project managers and staff to understand what participation means for differ-
ent stakeholders in the M&E process, and how this can be better managed. Appropriate plan-
ning of participation is important so as not to overburden stakeholders, or alternatively, to miss
out on vital information and opportunities for analysing and acting on it in a participatory way.
Exercise
Ask for examples of the benefits of adopting a participatory approach to M&E.Ask for examples
where participation of stakeholders can be negative.
Display a familiar example of a project matrix.
Present the different phases of the M&E process:
Develop the plan.
Gather the information.
Analyse the information.
Act on the analysis.
Share the learning.
Develop the matrix on page 32 of the handbook interactively, on a white board or flipchart paper.
Ask these questions during this process:
Who should participate at the different points in the M&E process?
What can they contribute?
How will it benefit them?
Will they be end-users of the information gathered?
Will specialised technical capabilities be needed?
What can be done to ensure maximum participation of different stakeholders?
When will the different phases in the M&E process happen?
Is specific information needed for regular management meetings?
Is the sequencing of different activities important?
Are there set review dates?
When are informants available?
Are project staff available to carry out M&E work?
Discuss participation in the M&E process using this stimulus question: ‘What are the management
responsibilities for ensuring participatory M&E?’
116 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Theme 4—The action learning cycle
Learning from experience is the desired outcome of our ongoing work, and lies at the heart
of a good M&E system. This learning does not happen automatically. Rather it requires that we
take some time to look back, to ask important questions, to make meaning of the data gathered
through reflection, to plan to make changes in our practice and then to implement these chang-
es. The action learning cycle is one model of learning that provides a set of steps to ensure that
learning does take place as a result of our experience.
Exercise
It is often said that ‘we learn from our experience’. Discuss this with the group and then request
the individual or group to identify ‘a slice of experience’, or ‘an incident in time’, that is common to
all in the group. It must arise from their work situation.
Ask the individual or group to write down a description of what happened, and then share it.
Collaboratively build a description of the incident on a flipchart.
Pose reflection-type questions, referring to ‘the action learning cycle’ on page 37 of the handbook.
Again, note the answers on the flipchart.
Facilitate a conversation about what insight and understanding can be drawn from what happened.
Ask individuals to draw up a plan to act on the insights gained.
Ask individuals to describe the planned actions to one another.
Present the diagram of the action learning cycle. Relate it to the steps just followed and to the M&E
process.
Monitoring and evaluation: tools for biodiversity conservation and development projects 117
Section 2: LEARNING
This section of the handbook further develops the conceptual and practical aspects of learning
within an M&E context.
e Theme |I—Learning concepts
Understanding how learning happens and exploring some learning concepts helps us to be
more conscious of the way we think and process information received via our different senses.
These patterns of processing could either enhance or limit learning, and so impact on our moti-
vation to change and develop our practice in the field.
A first step in encouraging our openness to ‘learning through doing’ is to understand our men-
tal models.
Exercise
&
y
‘SS
Display a simple picture of the outline of two people facing one another. Inside each of their heads
is a completely different shape—one has a whirl shape, the other a daisy shape with petals attached
to a clear centre.A common thought bubble reads, “You think funny!’ Have a conversation about
how different thought patterns could help or hinder learning and change.
Present a brief outline of Single-loop learning and generate some examples from joint experience.
Present a brief outline of Double-loop learning and generate some examples from joint experience.
Display two complex diagrams. One must be an example of known subject material such as an
endangered plant species, and the other unknown material such as a combustion engine.Ask the
individual or group to jot down their thought responses to each of the two diagrams. Write up the
two sets of thoughts on separate flipcharts.
Make a brief presentation on Patterning and Puzzling, relating each to responses in the previous
step.
Discuss how to go about facilitating learning in the project team. Finally, allow some time for
individuals to consolidate their understanding by drawing up a set of personal facilitation guidelines.
.) ,
Ra eo .
iy
\
118 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Theme 2—Tools for learning in the project team
The project manager also has to be the leader of learning in the project team. Being familiar
with, and having the confidence to use, a variety of learning tools makes it easier for project
managers to facilitate team learning and development.
Exercise
Request project manager/s to relax, close their eyes and imagine that they are about to learn
about some amazing new phenomenon in the environment.Ask some leading questions to help the
imaginative journey. Where are you? What is happening around you? What do you see? What are
you thinking? What is this amazing new thing?
Ask them to open their eyes.
Provide a sheet of paper and a set of oil pastels.Ask them to draw the phenomenon. Ask them to
work quickly and draw what comes to mind immediately rather than a detailed, precise picture.
Invite the sharing of the pictures, and the exchange of comments.
Then ask, What inspired and informed the ‘thing’ or phenomenon you imagined? What helped you
to imagine this? What in your background (your discipline, experience, skills) led you to imagining
this phenomenon?
Open up a discussion about how, consciously or unconsciously, we draw on a particular discipline
or field of expertise to notice our environment, imagine new things and understand the world.
Ask manager/s to reflect on the steps above and then list the different concepts, tools and
frameworks they used—or would like to have used—to enhance the learning experience. Display
the list/s and note common and unique tools listed. Use coloured markers to highlight the
differences.
Make a brief presentation based on the information found on page 43, ‘Using case studies to learn
about your conservation practice’.
Brainstorm a list of different learning tools.Add these to the list if they were not mentioned:
Peer assistance.
Debriefing.
Team meetings and workshops.
Bench marking.
Invite a brief description of these tools and refer participants to the notes on pages 45 and 46.
Allow time for managers to talk together to clarify opportunities to use the different team learning
tools in the future.
Conduct an ‘after action review’ of this capacity-building session and conclude by referring to the
notes on page 46.
e Theme 3—How to encourage learning
It is important to identify and understand the different factors that encourage learning and to
ensure that they are in place in the project team. It is just as important to know the blockages
to learning, and to have the capacity to clear or reduce the impact of these hindering factors.
The busy nature of conservation work can make it difficult to find the time to stop and learn.
So while we often go through the technical steps required to monitor and evaluate project
progress, individual and team learning can get neglected.
Monitoring and evaluation: tools for biodiversity conservation and development projects 119
Exercise
Make conscious what drives our learning.
Request project manager/s to reflect individually and identify for themselves the internal and
external drivers of their learning. Out of this personal reflection, ask them to further consider what
they think drives their team members’ learning.
Factors that encourage learning.
Make a brief presentation, drawing from the notes on page 50, ‘Building a learning environment’. Ask
project manager/s to assess which of these factors that encourage learning exist in the project/s
they manage.
How to ensure that the six enabling factors are present in the project team.
Ask what steps have to be taken to strengthen these factors.Allow time for individual work. Share
the steps and refine a set for each project manager.
Overcoming the barriers to learning.
Hold a conversation about ways to overcome the two barriers to learning (strong negative
emotions and a bias-for-action).
Action steps to overcome barriers to learning.
Invite individual/s to work on action steps that they, as project managers, could take to overcome
barriers to learning.
120 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 3: GATHERING THE INFORMATION
e Theme |—Different types of information
Information is the fuel that drives project M&E systems. It is important to gather the informa-
tion that is needed to feed into timely and relevant decision-making, reporting and learning. The
project plans should specify what should be monitored through the information gathering proc-
ess and the sources, or means of verification (MOYs), of this information.
Different types of information are needed to measure objectives at the level of outcomes and
outputs, as well as to track what is happening to the assumptions.
Exercise
Information relating to outcomes.
S
Present the characteristics of this information and provide some examples of different impact
reports. Hold a conversation about what is required to gather this kind of information.
Information relating to outputs.
Present the characteristics of this information, then discuss some of the problems encountered
with internal documentation, and possible solutions to this. Take into account issues of accuracy,
lack of uniformity, and the challenge of keeping a balance between quantitative and qualitative
information.
Information relating to assumptions.
Explore the importance of information that takes into account the presence or absence of external
factors on which the success of the project is dependent. Invite project managers and staff to give
examples of projects suffering the consequences of being unaware of changes in external factors.
Brainstorm ways to gather the relevant information and possibly influence the environment.
Qualitative and quantitative information.
Call for examples of qualitative information and list these on a flipchart. Repeat the exercise for
quantitative information.Ask participants to provide their definitions of both types of information.
Hold a conversation about the pros and cons of these different types of information.
Primary and secondary information.
Present the characteristics of both. See if there are any examples from their own experience that
participants can identify. Tabulate advantages and disadvantages of both and then explore ways to
overcome the challenges.
e Theme 2—Techniques for gathering information
There are a wide variety of techniques to gather information and it is important to expand the
project manager/s repertoire.A clear set of criteria to help select the most appropriate tech-
nique for a specific monitoring task will help them to make decisions about which techniques
to use in which circumstances. Each discipline has its own specific set of tools and techniques.
Draw attention to this, highlighting that the tools in this section of the handbook, and in the
handbook in general, come from the social sciences, and some specifically from the tradition of
participatory learning and action (PLA).
Monitoring and evaluation: tools for biodiversity conservation and development projects 121
Exercise
Favoured and familiar information gathering techniques.
Request project manager/s to identify the three techniques they favour and use most often. Ask
them to develop a brief set of pros and cons for each. Briefly discuss these ideas.
Expanding the repertoire.
Present a list of 15 different techniques, giving a brief description of each. (Take note of which
techniques have already been explored in the previous step and adjust the presentation.)
Selecting the appropriate technique for the task.
Present or generate a number of monitoring task scenarios and request the project manager/s to
select appropriate techniques from the list, using the selection criteria on page 57 of the handbook
(Timeline, Relevance and Cost-effectiveness).
Increasing understanding of different techniques.
Request project manager/s to take time to read through ‘A toolbox of techniques for gathering
information’ on pages 59 to 62. Divide the list into appropriate sections and allocate to small
groups. If working with an individual, the different sections would be worked with according to a
schedule. For each section:
» Discuss the methods
» Develop a set of pros and cons for each
» Use the monitoring scenarios from Step 3 as a basis to design an example for each of the meth-
ods
» Write up the design on a chart
» Share designs and invite feedback and suggested improvements.
Summarising.
Request project manager/s to read the summary of ‘Techniques for gathering information’ on page
58 and add to ‘Other factors to consider ’, based on their experience, and new insights gained.
122 Monitoring and evaluation: tools for biodiversity conservation and development projects
e Theme 3—Information management systems
In some projects, the act of gathering monitoring information becomes part of the habitual
pattern of the work process, and its purpose and further use is not considered, or gets
forgotten. Dusty piles of evaluation questionnaires administered after a specific intervention are
a common sight in many project offices. It is important for a coherent information management
system to be designed and put in place.
Exercise
Improving systems already in place.
Ask project manager/s to reflect on the project they manage:
* Identify all the monitoring processes that gather information at all the levels: Outcomes level,
Outputs level, Activities level, Input/Resource usage level
In what form is this information collected?
Who uses it and for what purpose?
How do all the pieces link up?
How is it stored and accessed when needed?
What improvements are needed to the system?
What does managing the information system involve?
Present a set of issues that need to be managed (draw from the notes on pages 63 and 64 of the
handbook) and then request managers to suggest what management interventions are required.
Exploring issues of quality of information.
How does one ensure the quality of information gathered? A series of small groups can work with
these issues below, and views collected in a plenary session. (Draw on notes on pages 64 and 65 of
the handbook).
Accuracy.
Representative of the whole picture (sampling).
Overcoming potential bias.
Comparability.
Consistency.
Focus on storage and retrieval systems.
Why is a storage and retrieval system important? Present key reasons:
= Comparison purposes.
= Impact studies.
= Future needs analyses.
Discuss the pros and cons of different storage and retrieval systems.An expert in the field of
information storage and retrieval could be invited to be a resource in this session.
Design of an information management system.
Using an example of a project plan, request manager/s to design an information management system
for the project. Share designs and comment.
Monitoring and evaluation: tools for biodiversity conservation and development projects 123
Section 4: ANALYSING THE INFORMATION
e Theme I—Using questions for analysis
Questions provide a critical starting point for analysing information. Being able to formulate
good, insightful questions is a key skill for project managers.
Exercise
Ask project managers and/or staff to imagine that they are someone coming from outside to
evaluate the project.
Divide them up into three groups: outcome objectives, output objectives and assumptions/enabling
factors.
Invite each group to brainstorm the types of questions that could be asked to establish progress
towards achieving these objectives, and to understand what the contributing factors are. Similarly,
what questions need to be asked to assess assumptions?
e Theme 2—Classifying evaluations
There are many different ways to classify and understand evaluations. One simple distinction is
that of external, internal and self-evaluation. Beyond this, there are further categories and clas-
sifications. It is a good idea for project managers to be familiar with the range of formal evalua-
tions available, and the terms and concepts that are used.
Exercise
4
AY
|. Share the different types of evaluations listed on page 76 of the handbook, and check understanding.
2. Ask whether their experience of these evaluations has been as external, internal or self-evaluation.
Make a list of which experiences fall under which category. Note if their experiences predominantly
fall into one category.
Based on these three categories, ask participants to brainstorm the different circumstances that are
best suited to external, internal and self-evaluation. Because evaluations are predominantly external,
push them to consider those situations in which it would be valuable to conduct internal and self-
evaluations.
e Theme 3—Contracting external evaluators
The process of contracting consultants and in this case, external evaluators, is something that
project managers in the conservation sector are frequently faced with. It can be a complex, and
sometimes fraught, process if it is not carefully thought through.
There are a number of aspects of the contracting process that need to be managed. These
include:
« Being clear on the objectives of the evaluation.
« Identifying a suitable consultant or team of consultants.
= Getting buy-in to the evaluation from internal and external stakeholders.
124 Monitoring and evaluation: tools for biodiversity conservation and development projects
* Agreeing on the approach and methodology for the evaluation.
* Managing the budget.
«= Managing the time frames.
* Managing the consultants.
Exercise
Debrief project managers and project staff on their previous experience of evaluations. This helps
them to clarify what works and what does not work for them, and how they might like to do things
differently in the future.
The ‘Sample terms of reference for an external evaluators’ on page 78 of the handbook provides a
very thorough example of the different elements to consider.
2. Part of the contracting process is developing this TOR collaboratively with project staff and
other important stakeholders. Assist the project manager in designing an inclusive process for the
development of a TOR.
Monitoring and evaluation: tools for biodiversity conservation and development projects 125
Section 5: ACTING ON THE ANALYSIS
e Theme |I—Identifying recommendations and change
If an external evaluation has been conducted, then traditionally the recommendations for the
project are generated by the external evaluators, with varying degrees of input from project
staff. Increasingly, evaluations are being held and managed internally by the project itself. This
provides an opportunity for project staff to really own and understand the issues that they are
being faced with (internally and externally) and to generate their own solutions to these issues.
Ensuring this kind of participatory approach to the evaluation process requires building it into
the TOR (see Section 4 of the handbook). It also requires that project teams have the capacity
to generate ideas, recommendations and solutions collectively.
Exercise
Drawing on the simple framework on page 84, ‘Keep doing? Stop doing? Start doing?’, encourage the
project manager to facilitate a discussion with project staff to generate some ideas and solutions to
a particular problem the project is facing.
Debrief the project manager after the team session, identifying what they did well and where they
require further support.
e Theme 2—Change and resistance
It helps for project managers to know that resistance is a normal and healthy part of the proc-
ess of introducing and making change.As project managers become better able to identify
resistance and work with it, their efforts at making change in their project will become more
effective.
Exercise
Using the resource on page 89 of the handbook, ‘Why resistance matters’, ask project managers to
generate their own understanding of the reasons for, and source of, resistance.
Work with them to identify and design a strategy for dealing with a significant form of resistance
that they are encountering in their project.
126 Monitoring and evaluation: tools for biodiversity conservation and development projects
Section 6: SHARING THE LEARNING
e Theme I—An expanded view of accountability
Accountability is often seen only in terms of reporting to donors. In this handbook, we have
tried to broaden people’s notion of accountability. For example on page 94 in the handbook,
we include not only upward accountability in our definition, but also downward, horizontal and
inward accountability.
Exercise
Share the definition of accountability (page 94 of the handbook) with project staff and discuss the
implications of this kind of multiple accountability.
S
Draw out what excites people about this, as well as what some of their fears and concerns might be.
Another practice that we have encouraged in this handbook in relation to accountability is to
link it to learning—that we are accountable when we share the learning that the project has
generated through its experiences. However, accountability and learning can sometimes be at
odds with each other, particularly when accountability is equated with reporting. It becomes
more difficult to share the mistakes (what did not work) and then the facts go underground.
Exercise
Ask project staff how they would like to make the link between learning and accountability. What do
they think will work in their context or their environment? Where can they take some risks and be
more radical in their approach?
S
e Theme 2—Making reporting accessible
Instead of seeing reporting as a chore, it can be seen as a valuable opportunity to create space
for reflection—a chance to really come to grips with the challenges and opportunities of the
project. However, to have the space and confidence to view reporting in this way means that
project managers and project staff need to understand the basics of reporting.
Exercise
In Section 6 of the handbook, we provide a basic framework for progress reports (page 99) and we
unpack some common donor terminology (page 101). Use this material to open up a discussion
with participants about their current reporting requirements. Establish where they are confident,
and where they require further support.
Agree on steps that can be taken to build their skills and capacity, and identify who can assist in this
process, by when.
Finally, the exercise on page 104, ‘Building a matrix of your project’s reporting requirements’,
can also be used to streamline the project’s reporting activities.
LA)
ab =
i “
Demco, Inc. 38-293
MCZ ERNST MAYR LIBRARY
wi iin
Biodiversity Series
. SANBI Biodiversity Series
a
—
. Conserving South Africa’s plants: a South African response to the Global Strategy for
Plant Conservation. 2006. C.K. Willis (ed.). ISBN 1-919976-28-0.
2. The Convention on Biological Diversity: biodiversity, access and benefit-sharing. A
i resource for teachers (Grades 10-12). 2006. A. Solomon & P. le Grange.
7 ISBN 1-919976-30-2.
3. The Convention on Biological Diversity: biodiversity, access and benefit-sharing. A
resource for learners (Grades 10-12). 2006. A. Solomon & PR. le Grange.
ISBN 1-919976-31-0.
4. Fynbos Fynmense: people making biodiversity work. 2006. A. Ashwell, T. Sandwith,
M. Barnett, A. Parker & F Wisani. ISBN 1-919976-29-9.
5. A plan for phylogenetic studies of southern African reptiles: proceedings of a workshop
held at Kirstenbosch, February 2006. 2006. W.R. Branch, K.A. Tolley, M. Cunningham,
A.M. Bauer, G. Alexander, J.A. Harrison, A.A. Turner & M.F. Bates.
ISBN 1-919976-33-7.
6. Invasive alien flora and fauna in South Africa: expertise and bibliography. 2007. C.F
Musil & |.A.W. Macdonald. ISBN 978-1-919976-35-8.
™N
. Project planning: tools for biodiversity conservation and development projects. 2007.
Cape Action for People and the Environment. ISBN 978-1-919976-37-2.
8. Bird Checklist for South Africa’s National Botanical Gardens. 2008. C.K. Willis,
O.E. Curtis & M.D. Anderson (compilers). ISBN 978-1-919976-41-9.
9. Guidelines for Offshore Marine Protected Areas in South Africa. 2008. K. Sink &
C. Attwood. ISBN 978-1-919976-43-3.
10. User profiles for the South African offshore environment. 2008. L. Atkinson & K. Sink.
ISBN 978-1-919976-46-4.
11. Monitoring and evaluation: tools for biodiversity conservation and development
projects. 2008. Cape Action for People and the Environment.
ISBN 978-1-919976-47-1.
ont.
Anic al publications by the South African National Biodiversity Institute
journal of botanical research.
eg
a series of occasional publications on the southern African flora and
ray =
QO Flowering Plants of Africa, a serial presenting colour plates of African plants
__ with accompanying text.
OQ Flora of southern Africa (FSA), taxonomic treatises on the flora of South
: Africa, Lesotho, Swaziland, Namibia and Botswana. Contributions also appear
in Bothalia.
Enquiries
SANBI Bookshop, Private Bag X101, Pretoria, 0001 South Africa.
Tel.: +27 12 843-5000
Fax: +27 12 804-3211
e-mail: bookshop@sanbi.org
Website: www.sanbi.org.
4q
‘
ISBN 978-1-91997b-47-1
AM |
976471")
G L ) B A L national
biodiversity
institute
SAN BI
ENVIRONMENT
FACILITY PARTNERSHIP FUND
Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Books to Borrow
Open Library
TV News
Understanding 9/11