Any kind of performance evaluation requires the collection of relevant data. In the following, some generic aspects that are relevant for data collection and evaluation
Figure 3.2: MCDA approach (diagram adapted from Linkov & Seager 2011)
are discussed. Full details of the data collection and evaluation for the case study are provided in Section 4.2.
3.2.1 Data Sources
In order to make use of all available information, a variety of data sources are sought that provide different types of inputs for the analysis. These include industry and literature data, complemented with expert elicitation.
Publicly available data Depending on the context, there may already be a rich body of literature data available, which can provide a good starting point for the analysis. In that case it is important to filter the data for the most rele-vant aspects that are applicable to the decision problem. Moreover, the quality and reliability of the data sources has to be ensured. For established technology that can be done, for instance, by consulting industry codes and standards. For more research oriented studies, publications in high-quality peer-reviewed jour-nals provide a valuable source. If data from different sources are available, they should be compared thoroughly and checked for consistency. Publicly available data can be very useful, however it may not be specific enough to be applicable to a case-specific situation.
Expert elicitation For more specific and/or technical aspects, expert elicita-tion offers a highly valuable source of informaelicita-tion. The data reliability, however,
depends on the level of expertise and issues related to subjectivity may arise. In the context of material selection for instance, experts could be biased towards ma-terials they are more familiar with. Therefore, evaluation of confidence alongside expert opinion is a sensible addition. Several methods exist for expert elicitation, such as surveys, interviews or different panel methods, which can be structured and targeted in different ways (Bouyssou et al. 2006). There are formal techniques for expert elicitation, for example the Delphi technique (Dalkey et al. 1969). In any case, breaking down the problem and seeking expert judgement in an area that is within their field of expertise, facilitates the process for the expert and increases the chances of obtaining reliable information.
3.2.2 Types of Data
Data will be available in different types and formats, which imply different con-straints and may require specific types of treatment. Some types of data and their implications are discussed in the following.
Quantitative and qualitative data Depending on the source, different data formats are common. The above-mentioned literature and industry data is gen-erally published as quantitative data, for example cost data, material properties, etc. Expert judgement can be provided in qualitative or quantitative format, again depending on the context and expected level of detail.
Qualitative expert judgement could be for example for comparing different op-tions, such as: ‘Option A is a bit better than option B on criterion number 1, but option B is slightly better than A on criterion number 2’. In such a case, qual-itative data can be transferred into semi-quantqual-itative data, by translating those judgements into a scaling system. When doing so, transparency about the scales needs to be ensured, and ideally the confidence in such judgements should be cap-tured alongside. The most common approach is to apply a point scoring system, where each qualitative judgement is translated into a certain score. Confidence levels can be accounted for through using a probabilistic or a fuzzy approach.
Bayesian methods may be used to ‘combine’ data from different sources such as expert judgement and historical data (ASME 2003).
Crisp and fuzzy data Using fuzzy numbers (as introduced in Section 1.1.2.3) helps to account for vagueness and imprecision in data. Instead of using a specific point estimate (= crisp data), values are provided as triangular or trapezoidal fuzzy numbers, which represent the degree of membership. The width of the triangle or trapezoid reflects the expected spread, the wider it is, the higher the associated variability and uncertainty. Fuzzy approaches require a good method-ological understanding by the user.
Deterministic and probabilistic data Another option is to introduce prob-abilistic approaches to account for variability and uncertainty. If input estimates can be provided probabilistically, this can lead to better appreciation about the most likely range of results. Probabilistic models are more complex than deter-ministic ones, so there may be a trade-off between additional efforts spent and the benefit from enhancing the results. In the case of life cycle approaches, where uncertainty about future predictions is significant, probabilistic approaches can be suitable to account for uncertainty and get a better appreciation about the potentially achievable ranges.
3.2.3 Limitations in Data
Gaps in data Even if there were no restrictions on sharing and accessibility of data, there would still be gaps and limitations on the data available, because of the complexity and variability of any natural or technical system. Additionally, in practice, issues with data availability and confidentiality often exist, when evaluating materials or other technological solutions on a life cycle basis. It may be required to estimate how large the influence of these data gaps on the results can be in order to appropriately highlight and communicate the gaps and their potential impacts.
Uncertainty in data Due to the long-term perspective of life cycle approaches, which involve forecasts into the far future, high degrees of uncertainty are un-avoidable. As introduced in Section 2.1.3, uncertainty can be classified into the aleatory and the epistemic type, with the former related to lack of knowledge and the latter related to the natural variability occurring in any system. Understand-ing the different sources of uncertainty allows for a better understandUnderstand-ing of the problem and targeting further efforts effectively. This is discussed in more detail in Section 3.4.
Data manipulation If data is provided in quantitative format, it is often per-ceived as more accurate, even more so depending on the level of detail presented.
This may also lead to manipulations, by presentating data in a specific format, e.g. as ratios or absolute values. Moreover, a high level of detail and precision in the values (e.g. by providing many decimal places) is generally associated with a high level of accuracy. For example the statements
• ‘Improvements of 1/3 are achievable.’ or
• ‘Improvements of 33.33 % are achievable.’
leave the reader with different impressions about the accuracy of the data, even though the underlying value is the same. Other enlightening examples of data manipulation in the context of material evaluation were presented by Allwood et al. (2012). It is therefore important to provide an appropriate level of detail, which reflects the confidence in the underlying data and the respective accuracy and precision of the data.