A key element of the CGIAR Research Program on Forests, Trees and Agroforestry (FTA) is Monitoring, Evaluation and Impact Assessment. Behind these unwieldy words sits a simple truth: we can learn from our mistakes or from things that go right. But to learn, we need to know what went wrong – or well – and why. This is where impact assessments come in. Daniel Suryadarma, Senior Scientist at the Center for International Forestry Research (CIFOR), was in charge of monitoring FTA research projects until March 2016. We asked him…

…how are FTA research programs monitored, evaluated and assessed?

Daniel Suryadarma: We have what we call the Planning, Monitoring and Learning Framework. The framework states that FTA is trying to contribute to complex systems with multiple actors, often opposing interests, and long time horizons.

In systems like these, it is very difficult – or dare I say impossible – to measure with certainty how much the contribution of FTA is to an observed change, either at the policy-making level or on the ground.

Therefore, we rely on a theory-based approach, where we use a tool called Theory of Change (ToC) to piece together how our activities and outputs will eventually lead to outcomes and impacts.

Tell us more about the Theory of Change. How does this work concretely?

Generically, such a Theory of Change works like this:

outputs are followed by an engagement strategy which then leads to intermediate outcomes and end of program outcomes and, eventually, impacts.

These impacts may vary depending on a particular project or activity, but the idea is that we have to be able to tell a coherent story of how our outputs, i.e. research results, will go through which channels – which we call impact pathways – to eventually reach the intended outcomes and impacts.

FTA has a Theory of Change that is aggregated from the Theories of each flagship. Also, each sizable research project, i.e. those worth over $500,000 has its own ToC. We work with scientists to determine these from the start of the project.

But it is important to note that these Theories are not set in stone – they should be revisited and adjusted as a project unfolds, because there may be unintended consequences, things that cannot be predicted.

An example is a biophysical research project found that the prevalence of trees improve water quality (as the roots act as filter, among others). A government took this finding and enacted a new policy that expelled all villagers living near the forests, so they don’t cut the trees down. This is an unintended consequence of the research, which merely wants to know the correlation between forest cover and water quality. Foresight is never 100%.

Once a project has a Theory of Change, the scientists can use it to guide them as they try to achieve their intended outcomes and impacts. This is where monitoring comes in. The scientists report to us when a particular target audience or outcome has been achieved, and we document it.

Finally, the ToC and the monitoring data will be utilized as the structure against which the project is evaluated and assessed. Usually, we mainly look at end of program outcomes as the benchmark of whether a project is successful or not.

But more importantly, a ToC will allow us to tell a story proving that the project indeed contributed to a particular outcome or impact. In addition, we also try to substantiate the size of our contribution.

In other words, we would like to know whether a particular outcome or impact for example, a new government policy on forests would have happened without our research.

The results of the evaluation will feed back into the design of new projects. This is what we call learning. So, there is a loop from planning to monitoring to learning to planning.

What are the key elements of the assessments? 

As described above, it is the Theory of Change, of course, and also evaluation questions and methods. We want to be as rigorous as possible, as that’s the only way we get meaningful enough results to learn from.

Are there consequences for the research and for implementing the results, depending on the assessment? What happens with the recommendations? 

Usually we leave it to the scientists to reflect on the evaluation results. I should say that this is where an approach called participatory evaluation, as opposed to an independent evaluation, becomes very useful. In a participatory evaluation, the scientists are involved closely from defining the research questions, finalizing the ToC, to reviewing the results. Learning takes place throughout these activities.

In an independent evaluation, for example, scientists are at an arms-length. I find that independent evaluations are not really useful for learning, as scientists who don’t feel like they are a part of the process will likely be defensive and reject the results or recommendations.

What were the most striking examples or success stories? 

FTA is actually quite successful in influencing policies and practices that will eventually result in better environmental and livelihood outcomes. For example

• Daniel Murdiyarso, CIFOR Principal Scientist, was one of the authors of the Indonesian National Strategy on REDD+;
• Research findings and tools from CIFOR’s Global Comparative Study on REDD+ are adopted by the Son La Forest Protection and Development Fund in Vietnam;
• CIFOR’s research findings on Payment for Ecosystem Services have been used by Vietnam’s Central Forest Protection and Development Fund and Ministry of Agriculture and Rural Development (MARD) to revise their policy and call for support from donors;
• FTA research gave input to the Letter of Intent between the Governments of Indonesia and Norway on reducing Indonesia’s Greenhouse Gas emissions. One idea that had been discussed for this letter was the possibility of an extensive tree-planting program rather than a moratorium on new logging concessions. The report Reducing Forestry Emissions in Indonesia showed that planting the number of trees needed to achieve emissions-reduction targets would require a land area twice the size of Indonesia. CIFOR was able to influence the way the Letter of Intent was drafted and negotiated because both parties trusted CIFOR’s science-based policy advice.

More examples can be found in the FTA brochure, at CIFOR’s Outcomes and Impact page, and in the assessment of the Global Comparative Study on REDD+. As most of CIFOR’s research is related to FTA, this is really an assessment of FTA.

Finally, we are currently developing about 10 Outcome Stories with scientists, showcasing the successes of various projects. They will be published shortly in CIFOR’s Forests News.