Reconocimiento lógico combinatorio de patrones

In this section, I will discuss the main research design challenges. Many studies about the impacts of forest certification have been conducted looking only at certified operations – particularly at the Forest Management Unit (FMU) level. Such studies employ a number of different methodologies and data-collection techniques. However, they have two important limitations: first, it is not possible to know if any impacts observed have come about because of certification or other factors as we do not have a “control group” of similar non-certified forest operations. Second, some of such studies are conducted in different and hardly comparable spatial and temporal contexts.8 _{Table 2.2, adapted from Romero et al. (2013) summarises some} of the possible solutions to overcome the most common methodological hurdles of studies addressing certification impacts. I will discuss each of those methods below.

8 _{For example, when analysing CARs across a range of different forest operations worldwide, some authors “put in}

the same bag” developed and developing countries. Also, as previously noted, CARs analysis mostly works for the FSC. Furthermore, collecting field data require significant expenses when conducted in large spatial contexts, and it is hard to tell if field differences concerning environmental impacts are due to different management.

METHOD DESCRIPTION LIMITATIONS

Experimental Randomly selected FMUs are allocated to the forest certification intervention.

Selection bias is likely because certification is voluntary. A

comparison based on the experimental approach is not feasible.

Quasi-experimental Because the certification treatment was not randomly allocated, a comparison group of uncertified FMUs needs to be constructed (counterfactual). The treatment and control groups should only differ in their certification status.

Comparison group construction is data- intensive and technically difficult. Approaches include matching

techniques (e.g., groups of certified and non-certified FMUs matched by factors that influence certification outcomes) and instrumental variables (e.g., correlated and easier-to-assess variables are used to infer impacts), among others.

Before–after Baseline data on key outcomes related to the certification intervention are measured and compared with data corresponding to the post-certification condition.

Data are often not available for all the variables before certification was granted for both treatment (i.e., certified) and control groups. Systematic review Intensive analyses of certified

FMUs, drawing on the history of the FMU and how the particular nature of the mechanisms and contextual factors produced change.

Time-consuming and knowledge- demanding: requires robust results of properly designed studies and thus fails to determine the integrated impacts of forest management certification unless available literature exists.

Expert judgment Assess the impacts of certification through compilation and synthesis of statements of people with profound knowledge of certification and the contexts in which forest management occurs.

Because forest management

certification is complex, this approach can be informative but may fail to capture the integrated effect of certification-driven changes and interactions with contextual factors.

Table 2.2 Potential advantages and pitfalls of approaches for understanding the impacts of certification. Source: adapted from Romero et al. (2013).

Romero et al. (2013) describe an experimental design, in which we randomly assign a treatment (the independent variable, that is, forest certification) among forest certification operations or FMUs as well as employing a control group that does not receive any forest certification intervention. Therefore, we can control for the aspects of the experimental setting, isolating the effects of the intervention (the dependant variables, that is, certification impacts). In principle, this method seems the most appropriate as we can measure the outcomes of the intervention (and subsequently, the effectiveness of forest certification) with much more accuracy. However, comparisons based on this approach rarely occur in real-life (Ferraro, 2009) because certification interventions are usually not randomly allocated by the researcher (Romero et al., 2013). Moreover, as Blackman and Naranjo (2012) point out, the risk of “positive self-selection” or “selection-bias” of participants9 _{can overestimate the impacts of the}

9 _{In practical terms, firms with the highest performance or closest to the requirements of the regime (forest}

intervention, which may be caused by other factors, undermining the internal validity10 _{of this} approach.

In Quasi-experimentaldesigns, the treatments (certification) are not randomly allocated, rather they reflect what occurs in real-world situations11_{; in other words, participants choose the} treatment by themselves (Cook et al., 1979). As in the case of experimental designs, the main limitation of this approach is the positive selection-bias. This limitation can be overcome with the creation of credible counterfactual12 _{cases, which need to be carefully selected (Greenstone}

and Gayer, 2009) among very similar comparison groups (here, certified and non-certified organizations) in a number of traits. For this reason, Blackman and Rivera (2010) and Blackman and Naranjo (2012) suggest the use of different statistical techniques to match13 _{organizations.} Thus, ‘the impact of certification is defined as the difference between actual outcome and counterfactual outcome’ (Blackman and Rivera, 2010). Some examples of this approach in forest certification are the studies conducted by Foster et al. (2008) assessing the FSC impact in post-harvested hardwood stands, De Lima et al. (2008) using paired sampling to assess the FSC socio-environmental impacts, Hagan et al. (2005) evaluating the effects of the SFI/FSC on biodiversity practices in US landowners, Dias et al. (2013) also evaluating the on-the-ground impacts of the FSC on biodiversity through using quantitative indicators (to estimate species richness), and the qualitative research of Kant and Brubacher (2008) about social impacts of the FSC on aboriginal communities in Canada.

The before-after approach considers a timeline in which the researcher can study the effect of the treatment in one or many organizations. In this case, the pre-certification outcomes

are the counterfactual outcomes, while the post-certification outcomes reflect the impact of the regime intervention. However, the effect of certification may be biased upwards, neglecting the presence of other confounding factors that may affect the outcomes even more significantly (Blackman and Rivera, 2010). Another limitation is that data cannot be available for all variables before the regime intervention (Romero et al., 2013), posing a practical and typical limitation. As an example, Hain and Ahas (2007) employed this design in their research on the effects of FSC in forest sustainable management in Estonia, through combining quantitative and qualitative methods of data collection.

10 _{The validity of a measure in social science is related with the precision of the research findings. Two types of}

validity are recognised: internal validity, which is related with the accuracy of the findings within a particular context and external validity, which accounts for the applicability of such findings to different contexts. See Miles and Huberman (1994).

11 _{They are also called natural experiments so they explore comparisons about what happens across different areas or}

over time; see Levy et al. (1995). In the same way, Young terms this research design as thought experiments; see Young (1999).

12 _{In other words, what would happen in the absence of the intervention (in this case, forest certification).}

13_{Propensity score matching techniques to control the selection bias effect have been used in sectors other than}

Systematic reviews are useful approaches (at least in exploratory studies) that can encourage, for example, the conduct of formal meta-analyses upon a number of particular issues and regions, and mixing different research designs depending of the available literature (Romero et al., 2013). Regarding these and most studies on forest certification impacts, Cashore and Auld (2012) criticise the narrow focus on very specific issues14 _{and the consequent} difficulty in extrapolation of their results. For instance, although Newsom and Hewitt (2005) focus on a huge variety of environmental, social and economic issues across a world review of certified operations, they may have overestimated the impacts of certification since they lack credible counterfactual cases (resting external validity to this study). Equally important are the limitations when conducting meta-analyses to assess the effectiveness of different schemes, so comparing numerous studies containing a range of different methodologies can be an insurmountable difficulty (Clark and Kozar, 2011). Overall, these studies provide an excellent starting point but they need to be complemented with other methodologies.

Lastly, Romero et al. (2013) put forward the expert judgement approach as a reliable method for assessing the impacts of certification. This comprises the evaluation of certification impacts by collecting and synthetizing the views of experts in forest management certification to usually reach a consensus among them around particular issues. But, as some (Woudenberg, 1991) have suggested, a high consensus among participants would not be necessarily related with a high accuracy of such a measure. In any case, this research strategy is usually complemented with other research designs (see for example Gomez-Zamalloa et al., 2011; Hain and Ahas, 2007).

Overall, each of the above proposed methodologies have certain strengths that can contribute to a thorough evaluation of the effects of certification. Except for experimental approaches, all of them can be applied to real-world scenarios. Nevertheless, they are limited in considering all the possibilities of such real-world contexts (for example, there are some cases in which we may find both after-research and counterfactual cases). That is the reason why the need to be complemented with different research designs.