Evaluation is a crucial component of the policy process (Dovers and Hussey, 2013) which can give insight to the relative merits of different approaches (Bottrill and Pressey, 2012;
Crabbé and Leroy, 2008; Mickwitz, 2003; Rossi et al., 2004; Vedung, 2000). Scriven (1991) provides the most general definition of evaluation, referring to it as “the process of determining the merit, worth, or value of something, or the product of that process”. Policy
13
instruments may be judged according to a wide range of criteria, including their cost, social or environmental outcomes, feasibility of implementation, equity implications, flexibility of use, information requirements and dependability (Dovers and Hussey, 2013; Gunningham and Young, 1997). In this thesis, I am primarily interested in the environmental outcomes delivered by policy responses to deforestation. I refer to policy effectiveness to describe whether a policy has capacity to or has delivered the environmental outcomes as per its stated objective.
In the context of the policy process, evaluation is most frequently referred to as the last
“stage” (Figure 1.1), wherein the process of implementing a policy and the outcomes delivered by the intervention are retrospectively assessed with respect to the policy objective.
However, “evaluation” may also be used to describe prospective analyses which aim to predict the potential opportunities, risks, and costs of an intervention during the policy formulation or design phase. I distinguish between these two broad approaches by describing the evaluation of the value or merit of a policy after it has been developed and implemented as ex post evaluation, and prospective policy analysis as ex ante evaluation (Crabbé and Leroy, 2008; Rossi et al., 2004). Since I consider both existing and emerging policy instruments for biodiversity conservation in this thesis, I draw on a range of ex ante and ex post evaluation methods.
Quantitative methods are commonly used for ex ante evaluation given their ability to predict the potential outcomes from policies under varying scenarios and assumptions. Examples of ex ante evaluation studies in the conservation literature include those which estimate the potential biodiversity benefits of protected area expansion (Venter et al., 2014; Watson et al., 2011) and reconfiguration (Fuller et al., 2010), quantify the carbon sequestration and biodiversity opportunities from forest governance interventions (Chapter Four, Carwardine et al., 2015; Evans et al., 2015), and determine the likelihood of biodiversity offset policies to achieve a ‘no net loss’ of biodiversity (Chapter Five, Bull et al., 2016; Gordon et al., 2011).
Of course, even the best designed ex ante evaluation cannot fully capture the realities of policy implementation, and so ex post evaluation is crucial for policy learning and
14
improvement. Both qualitative and quantitative methods may be used for ex post evaluation depending on its purpose, intended audience, the available data, and which component of the policy is being assessed (Margoluis et al., 2009; Rossi et al., 2004). Ferraro (2009) has argued that “the fundamental problem of evaluation” is in determining what would have occurred in the absence of the intervention i.e., the counterfactual. The counterfactual is frequently assumed, ignored or obfuscated in environmental policies (Maron et al., 2013; Maron et al., 2015), which makes ex-post evaluation extremely difficult. Establishing a counterfactual scenario is a pre-requisite for the use of experimental and quasi-experimental impact evaluation methods (Ferraro, 2009; Ferraro and Hanauer, 2014; Greenstone and Gayer, 2009;
White, 2009) which aim to quantify the causal link between a policy intervention and a variable of interest. However, even in cases where it may not be possible or desirable to conduct a strict impact evaluation, counterfactual thinking can assist in the design of non-experimental and qualitative evaluations which seek to establish what effect a policy intervention has had on an outcome relative to other contributing factors (Ferraro, 2009;
Margoluis et al., 2009; Rossi et al., 2004).
Recent debate in the conservation literature on the need for conservation policies and programs to be more frequently subject to ex-post evaluation has almost exclusively focussed on experimental and quasi-experimental designs (Margoluis et al., 2009). Some scholars have hypothesised that a of lack awareness and expertise in impact evaluation techniques is preventing more widespread knowledge of the effectiveness of conservation interventions (Baylis et al., 2016; Ferraro, 2009; Ferraro and Pattanayak, 2006; Miteva et al., 2012).
However, as outlined below, this argument obscures two key issues.
First, there is an assumption that the barriers to ex-post policy evaluation are purely technical, and are simply a matter of “better theory, better methods, and better data” (Miteva et al., 2012). In a recent survey, Curzon and Kontoleon (2016) found that awareness of impact evaluation amongst conservation practitioners and policymakers was extremely high, but more widespread use of such evaluation methods was largely prevented by financial and time constraints.
15
Figure 1.2 Illustration of an observed outcome under a policy or program (solid curve), and plausible counterfactual scenarios (dotted lines). The policy or program effect (‘impact’) is the difference between the observed outcome and the unobservable counterfactual. Adapted from Rossi et al.
(2004)
Keene and Pullin(2011)acknowledged that the impediments to an ‘effectiveness revolution’
in conservation are not only technical, but cultural and political. Organisations that are responsible for designing and implementing conservation policies and programs, whether they be government or non-government, can be reluctant to conduct or publish evaluations in the event a ‘failure’ leads to political or financial repercussions(Dovers and Hussey, 2013;
Keene and Pullin, 2011; Kleiman et al., 2000; Redford and Taber, 2000). Evaluation is also difficult when monitoring data is proprietary orotherwise inaccessible (Curtis et al., 1998;
Lindenmayer et al., 2017).
Second, the characterisation of impact evaluation techniques as ‘best practice’ overlooks the value and rigor of well-designed non-experimental and qualitative evaluations. Non-experimental quantitative evaluation designs, such as those which measure a variable of interest before and after an intervention, have less capacity to infer causality but can be used in situations where the available data is simply not amenable to a quasi-experimental design (Margoluis et al., 2009). InChapter Three, I use ahierarchicalbent-cable regression model to quantify the effect of regulatory policies on the deforestation rate in Australia. Despite
16
using the most up to date spatial imagery of deforestation events across the continent, the complexity and ubiquity of policies regulating the clearing of native vegetation across all Australian jurisdictions meant that even a synthetic control impact evaluation approach (Abadie et al., 2010, 2015) was not feasible. Our results demonstrate that the bent-cable model is a promising technique for detecting changes in the rate of deforestation in response to policy interventions introduced over time, and indicate that caution should be taken to avoid any premature claims of policy effectiveness prior to conducting an ex-post evaluation.
While quantitative evaluation methods can estimate what the impact of a policy intervention is to varying degrees of confidence, they cannot easily explain why or how these effects may have occurred. Qualitative evaluation approaches allow a far deeper examination of a particular case study, which can suggest why a policy intervention may or may not be working (Margoluis et al., 2009; Yin, 2009). Techniques such as interviewing, surveys and focus groups can reveal crucial information on the institutional and political context in which a policy is implemented (Blaikie, 2009; Hay, 2010; Yin, 2009), and provide evidence in the form of perceptions, knowledge and behaviour that can be used to improve conservation policy and programs (Bennett, 2016; Bennett et al., 2017; Moon and Blackman, 2014; St.
John et al., 2014). Particularly since conservation interventions are increasingly governed by multiple actors with varying motivations and objectives (Adger and Jordan, 2009; Lemos and Agrawal, 2006; Newell et al., 2012), qualitative methods can provide powerful insights into the whole policy process, and how these governance systems ultimately influence environmental outcomes. Chapters Six and Seven use qualitative methods to understand the governance ‘landscape’ of biodiversity offsetting in Australia, and the barriers and enablers to effective policy implementation. The lack of accessible data on the environmental outcomes being delivered by biodiversity offset policy in most jurisdictions worldwide (Brown et al., 2013; Lindenmayer et al., 2017; Maron et al., 2016; May et al., 2016;
Pawliczek and Sullivan, 2011) makes a quantitative evaluation of policy ‘impact’ close to impossible.
17