Diversity in an outlier detection ensemble can originate from a number of choices. An obvious –
but also the most challenging – choice is to choose different base algorithms. Other sources for diversity are parameterization (in particular parameters such as the number of neighbors and
the distance function), data preprocessing and projection (in particular, feature bagging [Bre96],
i.e. random selection of subspaces, also used in random forests [Bre01]) or the use of subsam- pling of the data set [Zim+13].
7.2.3.1 Diversity from Algorithms
Building an ensemble from different base algorithms appears to be the most appealing choice:
different algorithms can use different outlier models, and combining these results into a single outlier detection result should be able to find multiple kinds of outliers.
It turns out that it is non-trivial to combine the results of different algorithms, as already discussed in the previous Section 7.2.1. The early score normalization approaches in [GT06;
NAG10] are based on weak heuristics such asz-standardization for combining different results. The first successful approach was published in [Sch+12], and the improved score rescaling dis- cussed in Section 5.3 further improves these results.
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
(a) Mean Score
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (b) Maximum Score 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (c) Product, Equation 7.1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (d) Inverse-Product, Equation 7.2 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
(e) Bayesian, Equation 7.3,ϕ=.5
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (f ) Bayesian, Equation 7.3,ϕ=.1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 (g) Bayesian, Equation 7.3,ϕ=.01 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
(h) Bayesian Mean, Equation 7.4
7.2 Components of an Ensemble Method 155
7.2.3.2 Diversity from Parameters
An alternate source of diversity is to vary the parameters of an algorithm. For some parameters,
such askinkNN-based algorithms, one can expect that the score distributions remain similar, and thus simple variance-based rescalings such as HeDES [NAG10] are sufficient. Often, they
do not improve much over the baseline results, in particular when the scores follow different
distributions: HeDES rescaling can only make variance comparable, but not e.g. skewness.
Varying the distance function will provide some additional diversity; at least when distances
based on different paradigms are chosen. Merging these results will require careful score nor-
malization, as the distances returned by different measures will have different distributions. Furthermore (as seen before in Figure 5.21 and repeated in the experimental section in Fig-
ure 7.4), many distance functions belong to highly correlated families, and there often exists
a strong correlation even between different families. Due to the limited number of distance functions based on different paradigms, we can construct only small ensembles with a high
diversity: in our experiments we found that 20 popular distance functions can empirically be
grouped into 4-5 families that are highly correlated (see Section 7.4.1.2 and the block structure visible in Figure 7.4). Some approaches that could further increase diversity of distance func-
tions – such as using a randomized weighting – are equivalent to projecting the input data
differently. When using binary weights only, we obtain a standard technique to get diversity for ensemble methods known as feature bagging, which we will discuss next.
7.2.3.3 Diversity from Feature Bagging
An easy to use and powerful source of diversity is feature bagging [Bre96]. The benefits of this
approach are twofold: leaving away part of the data for each ensemble member yields rather
uncorrelated ensemble members (unless the dimensions are highly correlated), but it also helps with some of the problems with mining high-dimensional data (as discussed in Chapter 4): in
particular, the distance functions in the lower-dimensional space usually offer better contrast,
and ensemble members have a chance of skipping irrelevant dimensions. While some ensemble members may consist mostly of irrelevant attributes, they will likely also not find strong outliers
then, and will not find much agreement in the ensemble. Other ensemble members will however
consist of mostly relevant attributes and provide very clear results and be more likely to reach consensus. The optimal parameters for feature bagging will obviously depend on the amount
of redundant and noisy dimensions in the data set.
It is an interesting observation thatleaving away data is a good source for diversity in an en- semble, because it contradicts the intuition for building individual methods. However, in an
ensemble context, we redundantly use the data multiple times, so usually, no data will remain
unused in the end: if we construct an ensemble of25detectors, each using≈ 75%of the data, all data are used18.75times on average.
7.2.3.4 Diversity from Subsampling
If we performed “bagging” on the transposed matrix, instead of selecting some dimensions at
random, we select some instances of the data; a process known as subsampling and closely re- lated to bootstrap aggregating [Bre96]. The use of subsampling for outlier detection ensemble
construction was proposed in [Zim+13] as a source of diversity. Our own experiments with
subsampling for outlier detection date back to 2011, and the corresponding code was included in ELKI 0.5.0 [Ach+12] and used by [Zim+13]. However, we then did not use it in an ensemble
approach, but only used it as a baseline for evaluation approximate nearest neighbor search.
The important key contributions of [Zim+13] are the ensemble use case and an analysis of the theoretical properties. The main concern that stopped us from further pursuing this direction
was the observation that this emphasizes parameterization problems. When analyzing the per-
formance of subsampling approaches, algorithm parameters that depend on the data set size should be adjusted to prevent bias from the results coming from parameterization: as we have
seen before, most methods are rather sensitive to the choice of the neighborhood sizek. But when a method worked best withk = 5on a data set, when subsampling it tos = 10%, what would be the comparable value of k0to use on the subsample? Choosingk0 = s·k = 0.5is obviously not possible. Using the same k on the subsampled data set, however, may yield a result comparable to usingk/s(i.e.k0 = 50in the toy example) instead. In practice, for some methods such askNN Outlier, we found small values such as k = 1to work best. Since such parameters implicitly depend on the data set size, subsampling the data may have unpredictable
influence on the results, as seen in the experiments.
Similar problems can also arise in machine learning, which is why bootstrap aggregation does
not subsample the data set, but uses sampling with replacement to produce a data set of the same
size as the original data (and approximately the same class distribution). However, for outlier detection this is not beneficial, as we have highly unbalanced classes and need each single
object to be evaluated. For small classes, such as outliers, the errors introduced by bootstrap
aggregation will be much larger.
Some of the theoretical observations of [Zim+13] (based on earlier considerations on data bub-
bles in [Bre+01]) may only be useful for large values ofk: For small values ofk, the deviation of the truekNN distance from the expected value may become too large. For the 1-dimensional uniform distributionU[0; 1], thekth smallest value (also known as rank order statistic) is known to be Beta distributed: U(k) ∼ Beta(k, n + 1 − k). The mean of this Beta distribution is
k
k+n+1−k =
k
n+1. For thed-dimensional case the mean thus is
k
n+1
1/p
.1
In [Zim+13] the authors investigate the expected relative error of the distance to thek-nearest neighbor when subsampling. LetE[dk, n]be the expected distance to thekth nearest neighbor
1
7.2 Components of an Ensemble Method 157
innobjects. ThenE[dk, sn]/E[dk, n]is the relative change when subsamplingsnobjects:2
E[dk, sn] E[dk, n] = k sn+1 1/d k n+1 1/d = n+ 1 sn+ 1 1/d ≈n→∞ s−1/d . (7.5)
Roughly said, this proves that thek-distances are expected to grow by a factor ofs−1/d when subsampling. Of larger interest, however, is the variance of this term, as the variance translates into the diversity of the resulting ensemble. Rescaling the data set by a constant yields a similar
averagechange, but with 0 variance and thus does not introduce diversity at all. For the 1-dimensional case, the variance of this Beta distribution is known to be
Var[Beta(k, sn+ 1−k)] = k·(sn+ 1−k)
(k+sn+ 1−k)2(k+sn+ 1−k+ 1) =
k·(sn+ 1−k) (sn+ 1)2(sn+ 2).
If we now compute the relative variance (which is a squared measure, therefore we need to relate it to the squared estimate), we obtain (for 1-dimensional data)
Var[Beta(k, sn+ 1−k)] E[dk, n]2 = k·(sn+ 1−k) (sn+ 1)2(sn+ 2) k n+ 1 2 = (sn+ 1−k) (sn+ 1)2(sn+ 2) · (n+ 1)2 k ≈n→∞ sn3 s3n3k = 1 s2k, (7.6)
which translates to a standard deviation of 1/s
√
k for large n and 1-dimensional data. Un- fortunately, this equation does not as easily translate to multi-dimensional data as the mean.
Assuming a sampling rate ofs = 0.1and 1-dimensional data, thekNN distance is thus expected to increase by a factor of 10. For n = 100, the expected standard deviation is approximately
2.90, forn = 1000it is≈4.30with a limit forn→ ∞of10/
√
k ≈4.47. The high variance of this distribution prevents the subsampling approach from working well when used in a single-
run approach, for small sample sizess·ncombined with small values ofk. On the other hand, however, this variance is also beneficial to the ensemble use case discussed in [Zim+13]. The
errors introduced by different subsamples will be largely independent; but a low variance would
yield highly similar results; a high variance is likely a prerequisite for obtaining diversity. This may explain why sampling rates of10%were more beneficial in their experiments than higher sampling rates: doubling the sampling rate halves the variance.
Note that Equation 7.6 is not for the multi-dimensional case; unfortunately we cannot provide
a closed formula for theddimensional case. Experiments indicate that the formula will have a power term _1/dsimilar to the mean, meaning that subsampling will be much less effective for introducing diversity in high-dimensional data, due to the concentration of distances. Further-
more, the formal analysis assumes drawing new samples from the same uniform distribution
2
1 1.2 1.4 1.6 1.8 2 2.2 2.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
5NN distance sample / 5NN distance full
Sample size s n=100 (a) Dimensionalityd= 5 1 1.2 1.4 1.6 1.8 2 2.2 2.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
5NN distance sample / 5NN distance full
Sample size s
n=100
(b) Dimensionalityd= 27
Figure 7.2: Relative error of 5-nearest-neighbor distance,n= 100,s=.05. . .1.0.
instead of actually sampling the existing data (this is the reason why the expected variance for
s = 1 andk = 1 is not0). Also in practice, sampling will not affect every object with exactly the factor s. Only on average, s·n of the original nearest neighbors will remain in the sam- ple. For smallk, this will also contribute to diversity. Nevertheless, these equations show the general behaviour: diversity increases with lower sampling rates, but less so with high dimen- sional data. Figure 7.2a visualizes the quotient Equation 7.5 for k = 5 when sampling from
n= 100instances distributed uniformly in a 5-dimensional unit ball. We performed 1000 itera- tions and visualize the mean, standard deviation, minimum and maximum using whisker plots (compare to Figure 1 in [Zim+13]). On 27-dimensional data (we chose 27, because the data set
we will be using in the experimental section has 27 dimensions), the variance obtained this way
is substantially lower, as seen in Figure 7.2b.
In conclusion, there are multiple (overlapping and sometimes conflicting) effects happening
when using subsampling that make evaluation difficult. On one hand, the performance will (most obviously forkNN-Outlier) be more similar to that ofk =k0/s, which can be beneficial or problematic. On one hand, it improves scalability for large data sets, but on the other hand,
for data sets wherekNN-Outlier withk = 1works best, subsampling will usually deteriorate. In our experiments, LOF usually worked best withk ≈ 10, which explains why in [Zim+13] they had best results with a very low value ofk = 2. The other effect is that of variance, which allows ensemble approaches to work.