A common issue in DEA is that it does not discriminate between the efficient units (see chapter 2.4). Furthermore, it bases its efficiency estimates on a comparison of input- output levels of an individual DMU with those of a very small subset of efficient peers. In this respect, it can be highly sensitive to data swings at the individual DMU level (Thanassoulis, 1993). A frequently noted advantage of DEA is that it assumes complete
DMU no. Score actual projection CMII CommE CMII CommE 1 0.46 55768.45 20024.08 122188.38 43872.66 2 1.00 108125.49 34319.65 108125.49 34319.65 3 0.55 67911.92 8139.46 123946.48 18193.46 4 0.42 38616.88 5784.05 91380.01 29480.8 5 1.00 395736.06 40797.61 395736.06 40797.61
192
substitutability of inputs and outputs and therefore assigns individual weights to derive the highest performance score possible for each DMU. An issue that may arise from such an analysis is that some DMUs may be evaluated highly efficient, although they excel in some of the determined performance criteria and at the same time assign zero weights to others (R. R. Thomas et al., 1998).
Since the results of the DEA assessment shall be used for evaluating employee performance on a continuous basis, the acceptance, and thus the validity of the data, is of crucial importance. Therefore, a sensitivity analysis of the results was performed before processing them any further.
7.3.2.1 Parameters for sensitivity analysis
For sensitivity analysis three issues that are often discussed in DEA application (see chapters 2.4 and 2.5) were taken into account:
Unrestricted assignment of weights
A first important parameter for sensitivity analysis concerns issues that arise from the unrestricted assignment of weights. Besides the situation that employees may excel in some criteria only, it may occur, that weights assigned by DEA are not reflective of management´s strategic map (R. R. Thomas et al., 1998). On the contrary, the so called “hands-off” policy of DEA is frequently mentioned as a major advantage by both researchers and practitioners. After a discussion with the bank´s management it was agreed that the “hands-off” policy should generally be retained. However, it was decided to rerun a weight-restricted DEA for each stage to see whether there are “specialists” (employees whose high rating is based on only few criteria). It was agreed to use a rather moderate restriction with u,v ≥ 0.1. That is, each criterion had to be considered at least 10 percentage points in the calculation of the performance score. To assess whether the weights imposed had a significant impact on the efficiency rating of a certain DMU, Thanassoulis (1995) suggested to highlight cases were efficiency drops by 10 percentage points. As bank´s management and workers´ council felt this was too restrictive, it was decided to highlight cases were efficiency drops by 30 percentage points.
193
To detect data errors, it was agreed to remove one of the highly correlated variables (r > 0.75) and see whether the removal had a significant impact on the overall results. According to common statistical practice, a significance level of p ≤ 0.05 was determined. Thus, the impact would be assessed as significant, if the mean score dropped or increased by more than 5 percentage points. If there were more than two highly correlated variables the procedure needs to be repeated accordingly.
Identification of outliers
To discriminate between efficient units and identify outliers, several methods have been proposed in literature. For the case-study it was decided to apply the concept of superefficiency for classifying efficient units for two reasons (see chapter 2.4). First, the Anderson-Peterson model is more convenient to calculate as there is no cross tabulation to create. Second, the results are easier to interpret for practitioners. Since the superperformance score of a particular DMU measures how much the efficient frontier is shifted towards the origin by the removal of that DMU, superefficiency can be interpret as a measure of the DMU´s influence (Wagner et al., 2003). To assess whether a DMU is of high influence, Wagner, Shimshak et al. propose a limit of 2.00, which was adapted for the case study. In a second step, the actual influence is investigated by removing DMUs with a score larger than 2.00. This is done by removing only one DMU for each DEA run (e.g. if DMUs A and B have a score greater than 2.00, one DEA is run with only DMU A removed, and a second with only DMU b removed). Again, a significance level of p ≤ 0.05 was agreed to assess the impact of the DMU´s removal on the mean score. Thus, if the mean score drops or increases by more than five percentage points, the DMUs influence is significant. Although employees with a high superperformance score shall not be penalized for showing superior performance, it was agreed to remove them from the data set, if their influence was significant. The decision was based on the belief that otherwise it would be rather hard for all other employees to achieve efficiency.
194
Figure 34: Procedure for sensitivity analysis
Figure 35 displays the whole process of sensitivity analysis in detail. To evaluate the influence of each of the mentioned factors separately and independently, it is of crucial importance to restore the data set after each DEA run, as illustrated above.
195
7.3.2.2 Application of sensitivity analysis
The application and the results of the sensitivity analyses are illustrated in the following. The detailed results for each stage are provided in annex 2.
The table below holds the summary of results from the sensitivity analysis for TS1.