The following criteria were chosen to assess the effectiveness of the selected methods. The justifications for the selection of the criteria are discussed below by main criteria and the associated sub-criteria.
Under the dimension of scientific soundness, twelve sub-criteria were considered. These are described in order:
1. Sustainability Concept: The concept of sustainability needs to be well-defined for sustainability assessment (Pope et al., 2004; Zahm et al., 2008) and is usually based on the Triple Bottom Line approach (UN, 1987) or a principles-based approach (Gibson, 2006; Pinter et al., 2012; vanLoon et al., 2005). Due to many inherent limitations of the triple-bottom-line approach including ambiguity, principles-based approaches are more appropriate for concept development because they avoid these limitations (Pope et al., 2004). A well-defined concept of agricultural sustainability provides a strong basis for defining which indicators are needed for assessment (Sathaye et al., 2007; vanLoon et al., 2005). Assessment based on a well-defined concept can support the development of robust agricultural policy that in turn supports sustainability (Van Pham & Smith, 2014).
2. Methodological paradigms for the development of indicators: Agricultural sustainability indicators can be developed under two broad methodological paradigms: top-down
38
(expert-led) and bottom-up (community/stakeholders-based) approaches (Roy & Chan, 2012). In a top-down approach, experts select the set of indicators based on their expertise (Bossel, 1999), whereas in a bottom-up approach, the opinion of the stakeholders/community are considered in developing representative indicators of systems (Reed et al., 2006). Indicators can also be developed by involving both stakeholders and experts. In terms of indicator development, the approach that gets input from both stakeholders and experts is the most effective (Fraser et al., 2006; Reed et al., 2006). 3. Justification of indicator selection: It is important to understand the justification for the
selection of the indicators in order to understand and link them with agricultural sustainability. It is also important for transparency and replicability reasons (vanLoon et al., 2005).
4. Data sources for indicators: Agricultural sustainability indicators can be developed based on both primary and secondary data sources (Dantsis et al., 2010). These need to be technically sound, generate acceptable guidelines and standards and be subject to peer review (UN, 2014). Indicators that are developed based on primary data and validated by secondary information are most sound.
5. Use of qualitative and quantitative data to develop indicators: In agricultural sustainability there are many considerations such as good governance, labour rights and so forth that can be measured using qualitative indicators (FAO, 2012). An assessment system that can handle both qualitative and quantitative information is appropriate for sustainability assessment.
6. Ability to consider sustainability issues across scales in developing indicators: As agricultural sustainability is influenced by different issues across a spectrum of scales, including local, national and global (vanLoon et al., 2005), it is important to consider the
39
issues of the integration across scales and over time. Many policies, management programs and assessments for human-environment systems fail because they do not appropriately address issues across scales (MEA, 2005). Integrating different issues across spatial and temporal scales (one year or a series of years) can help to produce a more holistic picture of sustainability. This is different from the spatial applicability of the methodology as stated in criterion 12. This is related with sustainability issues across scale whereas criterion 12 is related with the applicability of assessment methods in different spatial scale (e.g., farm, local, nation and regional agricultural systems).
7. Validation of indicators: “An indicator will be validated if it is scientifically designed, if the information it supplies is relevant” (Bockstaller & Girardin, 2003:641). Validation helps to identify transparent indicators of ASA.
8. Reference values of indicators: Reference values describe the desired level of sustainability for each indicator (van Cauwenbergh et al., 2007). They can be based on legislative norms, scientific norms, or observations in the study areas (Sauvenier et al., 2006) and/or defined by stakeholders and experts. Reference values can also be applied to compare sustainability levels (Acosta-Alba & Van der Werf, 2011). “Reference values help to interpret the indicator value and may guide the evolution of a system towards an acceptable level defined in the objectives of the study. Reference values are requested by users, because they help to interpret the method’s results” (Acosta-Alba & Van der Werf, 2011:425). A reference value can act as a threshold value (Hrebicek et al., 2013).
9. Data normalization: Data normalization brings different indicator values into the same scale and facilitates comparison (Benini, 2012). “Whenever indicators in a dataset are incommensurate with each other, and/or have different measurement units, it is necessary
40
to bring these indicators to the same unit, to avoid adding up apples and pears and to help avoid dependence on the choice of measurement units” (Nardo et al., 2005:11).
10. Data aggregation: Aggregated indicators lead to an integrated and holistic approach to sustainability considering different dimensions of agricultural sustainability (Van Passel & Meul, 2012). Usually, the meaningful components and indicators are identified from each dimension of sustainability, then a single scoring system is applied to add indicators and to aggregate sustainability measures (Gafsi & Favreau, 2010).
11. Sensitivity analysis: Sensitivity analysis is “used to determine how different values of an independent variable will impact a particular dependent variable under a given set of assumptions” (Akasie, 2010:253). Sensitivity and uncertainty analysis play a fundamental role in increasing the quality and robustness of the answer provided by a sustainability assessment (Ciuffo et al., 2012:18). “Sensitivity analysis is performed for two reasons: robustness analysis, and ‘what-if’ analysis. Both approaches use perturbation of input values. ‘What-if’ analysis aims at pinpointing those inputs that affect output the most” (Information Resources Management Association, 2014:176). Sensitivity analysis helps decision makers formulate agricultural policy by assessing potential scenarios (Information Resources Management Association, 2014).
12. Spatial applicability: Spatial applicability is important to the extent that the method can be applied across scales (i.e., farm, local and regional). It will be much more appealing to policymakers and stakeholders if it can be applied in diverse agricultural systems across scales.
41
The main criterion of user-friendliness captures the extent to which the ASA method is flexible and easy to use. It includes graphic design, calculation (automation) and ease of assessment (De Mey et al., 2011). The following five sub-criteria were used to assess user-friendliness:
1. Learning dimension: The application of an ASA method itself is a learning experience since it deals with many issues (vanLoon et al., 2005). It is important that the method focuses on filling the gap in sustainability assessment and shows the steps towards utilization of the research findings.
2. Presentation of results: Results presented in a clear and multi-perspective manner (both graphical and numerical) are more attractive to users and stakeholders. Van Passel and Meul (2012) observed that results presented using visual tools are helpful and appropriate for farmers to understand farm sustainability, whereas policy makers benefited most from the numerical integration tools applied at farm to regional levels.
3. Available as software with video tutorials and with free access: Availability and free access to software and video help stakeholders implement the method, manage and analyze data, present the results and demonstrate how to use the methods. Software allows for fast, automatic calculation of huge data sets. It also allows various stakeholders to use the method. Availability of software can improve communication among wider stakeholders and policy makers.
4. Guidelines: User guidelines allow stakeholders to use the methods effectively, help in indicator development and aid in analysis and generation as well as the communication of results. Guidelines should clearly describe or lay out all the procedures for the method. 5. Certification procedure or advisory tool: ASA can be used for certification or as an advisory
42
tool will suggest how to improve agricultural systems through an analysis of management weaknesses (Hrebicek et al., 2013). Knowing whether it is a certification procedure or advisory tool aids in communicating the results.
These two main criteria and their associated sub-criteria are now applied to test eight methods that can be applied to ASA.