As discussed in Chapter 1 and 2, it can be challenging to quantify risk and envi-ronmental impacts in an objective and representative way. If resources are lim-ited and an extensive quantitative assessment is not feasible, a semi-quantitative assessment, where a qualitative expert evaluation is transferred into a scoring sys-tem, can provide an effective starting point to obtain a good understanding.
Therefore, a set of criteria, which is relevant with respect to the objectives, needs to be established. These are called Performance Indicators (PIs) and can be identified through a survey, brainstorming sessions or any other form of expert elicitation. PIs may be directly related to the fundamental objectives, or they could be relevant to the means objectives, which are also important, because they influence the fundamental objectives through other PIs. From the poten-tially large number of PIs a set of Key Performance Indicators (KPIs) needs to be established, which meets the requirements of representing the fundamental objectives in a way that is complete, concise, unambiguous, understandable and operational (Gregory et al. 2012). In the context of sustainability evaluations, the suggested KPIs are factors that are directly related to the EI and RA evaluation.
Accordingly, a set of environmental and risk related KPIs is established.
Subsequently, the KPIs need to be evaluated in comparison to the baseline op-tion. Since for some aspects even a small improvement in performance is quite significant in terms of their impact, and vice versa, the evaluation is split between Performance and Significance. Moreover, it is assessed how much any changes apply to the different life cycle stages. Consistent with the life cycle approach depictd in Figure 3.3, it is distinguished between
• Initial stage - including design, purchasing of raw material and production,
• Operation & maintenance stage - including general operation as well as maintenance actions, such as inspections, repair and replacement,
• End of life stage - considering potential recycling or else disposal of com-ponents.
Performance evaluation Quantitative and qualitative data are collected for the performance of the different options compared to the baseline. This is trans-ferred into a semi-quantitative assessment by assigning a performance score Pij, that provides the performance of the jth alternative on the ith KPI. The scaling of Pij depends on the context, the level of granularity required and personal pref-erences. For a relative assessment, the choice of a suitable baseline is required.
Since the cost factors are assessed in % compared to the baseline of 100 %, a score of 100 is likewise chosen for the baseline in order to enable comparison.
Subsequently, changes in KPIs need to be scored relative to this baseline. Since the goal is to minimise risks and environmental impacts, lower performance scores Pij are assigned for comparably higher performance and vice versa. The scoring system should be agreed at the beginning of the expert elicitation process, so that it is clear and transparent how qualitative evaluation will be translated into the performance score.
For the environmental impacts, the performance score PE,ij refers to how much the environmental impact is expected to change when choosing the alternative options compared to the baseline. If broken down into the different life cycle stages, a relative assessment based on available data and expert judgement is generally possible and can provide a reasonable starting point for highlighting the key aspects.
With regard to risk aspects, the performance score PR,ij can be understood as capturing the relative changes in the likelihood of health and safety risks. As opposed to the term PoF, which is generally expressed in a quantitative format, likelihood can also be expressed in qualitative or semi-quantitative format. In most cases, choosing a different alternative will influence the likelihood of failure or in other words the likelihood of the health and safety risk. The performance score for risk aspects PR,ij is designed to capture such changes.
An example for a scoring scale is provided in Table 3.2. In this example an
‘extremely worse’ performance would increase the score by 50 %, whereas an
‘extremely better performance’ would reduce the score by 50 %. For instance, it can be interpreted as: A reduction in GHG emissions by x % is valued as
‘slightly better’ in terms of environmental performance. For risk aspects, a certain material behaviour is valued as ‘extremely worse’ if it significantly increases the likelihood of health and safety risk. Table 3.2 is for demonstration purposes only.
It can be amended by using more or less categories and by scaling the attributed performance scores Pij as required. How confidence in the evaluation can be factored in, is discussed in Section 3.4.2.2
Table 3.2: Example of a performance scoring system around the baseline score of 100
Evaluation of significance In order to relate the performance on the different KPIs to the fundamental objectives of the decision problem, the significance of the changes in KPIs with regard to the fundamental objectives are evaluated.
This is done by assigning a significance score Si, for example as in Table 3.3.
Essentially, Si is a kind of weighting factor, which is applied to each performance score Pij.
Table 3.3: Significance of individual KPIs to health and safety risk and envi-ronmental performance
Similar to the performance scores, the scale can be amended as required. It has to be noted that the significance scores Si are assigned to the set of KPI independent of the performance of the different alternatives. For example, with
regard to environmental impacts, the significance of GHG emissions could be rated as high, compared to acidification potential that might be rated as medium.
In terms of health and safety risks the significance score is closely related to the CoF. For example, different factors that influence structural integrity could be assigned a very high or a very low rating of significance, depending on the context and the circumstances. For both cases, the significance score Si is set irrespective of the performance scores achieved. All different alternatives j have the same significance score Si per KPI, which therefore provides a measure of the relative importance of the different KPIs.
Calculation of final scores The total performance scores Etotalj for environ-mental impact and Rjtotal for health and safety risk for each alternative j are determined by multiplying the individual performance scores Pij with the signif-icance scores Si, normalising them and aggregating them for all KPIs according to Equation 3.8 and 3.9, respectively.
Etotalj =
Since Si is essentially a weighting factor, the normalised significance score Si0 is applied, which is calculated as
Si0 = Si Pn
i=1Si (3.10)
This can be done separately for each life cycle stage or by generating a total life cycle score by calculating the sum of the individual scores. If a total score for the life cycle is calculated, some weighting between the life cycle stages is required.
For more transparent results, it is recommended to keep the different life cycle stages separate at this point. Further details about weighting and aggregation are discussed in Section 3.5.