5 Discusión
5.3 El sentido de la educación emocional
This class of statistical learning methods is named after their strategy to look into the vicinity of the given event in input space. For classification the events of both classes are counted in this region and their numbers are compared. The class with the majority of events is the best guess for the class of the new event. For regression usually a (weighted) mean of the target values is calculated. Local density estimators differ in the way they decide whether a training event resembles the new one.
5.3.1
k-Nearest-Neighbours
This algorithm searches for the k training points with smallest distance to the new event. The size of the spherical region surrounding the new event is therefore (theoretically) adapted such that exactly k training points are taken into account (see figure 5.2). For classification the events in this region are counted for both classes, resulting in Ngood
(targety = 1) and Nbad (target y = 0) of the k events and the output is
out= Ngood k . (5.3) 1 1 1 1 1 1 0 0 0
Figure 5.2: An example for a 5-nearest- neighbours evaluation with result 35.
Regression
For regression the (weighted) mean target value of thek events is calculated:
out= 1 k k X i=1 wiyi ( X wi =k). (5.4)
5.3 Local Density Estimators 85 Parameters and Regularisation
A free parameter of thek-nearest-neighbours search iskwhich determines the overtraining behaviour. A very local, fine granularity decision is generated by a small k while a largek
means a very soft decision insensitive to very local properties.
Another important parameter is the metric with which the distances are calculated. One can, for example, insert scaling factors along each axis into a standard Euclidean metric so that
k~xk=
r X
γix2i. (5.5)
Execution Times and Variants
Ask-nearest-neighbours determines its output directly from the input data without build- ing a model no training time is required. For the evaluation of a new event the distances to all training points need to be checked and the smallestk distances need to be found. This makes evaluation times very long. Many strategies have been invented to deal with this problem, ranging from smart indexing schemes to algorithms creating a condensed version of the training set with less events but trying to keep all information [70].
Variants also emerge from the distance calculation. As mentioned above, the metric can be varied and the next section introduces the idea to use the general concept of kernels instead of a specific metric.
5.3.2
Kernel Methods
As a generalisation of the free choice of the metric in the k-nearest-neighbours search the concept of kernels is now introduced. A simple local density estimator is, for example,
Parzen’s window [71]: local density(~x) = 1 n 1 V n X i=1 φ Ã k~x−~xik λ ! φ(u) = ( 1 if |uj| ≤ 12∀j 0 else (5.6)
where the summation over φ simply counts the events within a region of volume V scaled byλ. Sincek~zk2 =~z·~z, any distance measure can be modified by exchanging the definition
of the dot product. A very general density estimation is therefore given by local density(~x) = 1 n n X i=1 K(~x, ~xi) (5.7) where the kernelK represents any strategy for distance measurement (and thus weighting) among all events. For example the Gaussian kernel
K(~x, ~xi) = 1 ³√ 2πλ´p exp − 1 2 X j µx j−xji λ ¶2 (5.8)
could have been used instead of Parzen’s window above. Like for Parzen’s window,λscales the size of the window (with soft borders in the kernel) around the point for which the density should be estimated.
86 5. Statistical Learning Methods Regression
If each label shows a real value instead of 0 and 1 for the two classes one does not compare the two local densities anymore but one weights the target value yi with the distance measurement resulting from the kernel:
out= 1 n n X i=1 K(~x, ~xi)yi. (5.9) Parameters and Regularisation
Any chosen kernel has its own parameters, some of which may need to be set manually for each dataset. Parameters like λabove should always be available because they control the overtraining behaviour: A small λ makes the window very narrow resulting in a classifier sensitive to very local density differences. A large λ will create a very smooth classifier which is only sensitive to more global density differences.
Execution Times and Variants
The same problem as for k-nearest-neighbours search applies here: In principle all train- ing events have to be processed. In the same way the different workarounds can be used. Variants clearly come from using different kernels, some more will be mentioned in sec- tion 5.4.3.
5.3.3
Range Search
To deal with the problem of high computing costs for large datasets (for each evaluation all training points need to be taken into account) range-searching [72] inserts all training points into a binary tree structure and needs to traverse this tree only partially for one evaluation. For this a box is placed around the position which should be evaluated and data points outside are ignored, see figure 5.3 for an illustration.
The output of the range search method for a given evaluation position is the fraction of the “good” training events lying inside the box:
out= Ngood
Ngood +Nbad. (5.10)
Regression
Like for the methods discussed so far in the case of regression the output is the (weighted) mean value out= 1 N N X i=1 wiyi ( X wi =N). (5.11)
Parameters and Regularisation
The crucial parameter in the range search method is the size of the surrounding region which should be looked at. If it is too small the statistics is bad and overtraining will be seen, if it is too large the performance will drop. To find the right size – in the most general case two (asymmetric) values for each input – is an involved task (compare section 7.2.7).
5.3 Local Density Estimators 87 1 2 2 1 1 2 1 x 1 2 3 4 5 6 1 2 3 4 5 6 < > < < > x x x x x 1 2 3 4 5 6 6 1 2 4 5 3 x
(a)
(b)
Figure 5.3: (a) The binary tree structure is built up by changing the coordinate which is used for ordering in every level. The events are filled into the tree as they are read in. Event 1 is read in first. The binary split on the first level is determined by thex1 coordinate, thus the events 2
and 3 are filled into the two branches of event 1 because 2 is found left of 1 and 3 right. Event 4 is left of 1 and is thus filled somewhere in the branch of event 2. The binary split on the second level is determined byx2 and event 4 is found to be below event 2. Event 5 is found right of event
1 and below event 3 while event 6 is right of event 1 and above event 3. (b) The search for the points inside the box can exclude whole branches of the tree by comparing the coordinates with the bounds of the box. The shown box is completely right of event 1 and can therefore exclude event 2 and the whole branch below it. Events 3 and 5 are found inside the box, event 6 has to be checked since the box is not completely below event 3.
Exection Times and Variants
Like k-nearest-neighbours search, range search is not either a model-based algorithm: The decision is directly derived from the training set. Therefore no training times exist. But evaluation can take quite long because the tree needs to be partially traversed for each evaluation position.
A modification of the basic algorithm may improve the performance: A slower but more effective way of setting a box size could be to use an adaptive box size. This means that the hyper-box surrounding any evaluation position is scaled to fit the local density of data points so that the number of training points in the box is of the same order for any evaluation position.
5.3.4
Naive Bayes
The naive Bayes classifier (also called maximum likelihood) [73] derives its decision – like Bayesian learning discussed in section 4.2 – from Bayes theorem
P(C|~x) = P(~x|C)P(C)
P(~x) (5.12)
which calculates the conditional probability P(C|~x) for class C (0 or 1) given the input vector ~x. Maximising P(C|~x) (finding the correct class) means maximising P(~x|C)P(C). The naive Bayes assumption isP(~x|C) = Q
iP(xi|C) (independentness) and thusP(C|~x)∝
P(C)Q
iP(xi|C). The probabilities P(xi|C) are estimated from histograms as described below. The prior class probability P(C) may also be estimated from the data. In the following however, they are assumed to be equal for both classes.
Two histograms per input are needed to estimate P(xi|0) and P(xi|1). These his- tograms are projections onto the axes xi (assumption of independentness) of the two
88 5. Statistical Learning Methods multidimensional sampling distributions given by the training data. By the projection process, however, the correlation between the input variables gets completely lost. This uncorrelated approach is frequently used, while the large amounts of data required for the correlated, not projected version commonly forbid its usage in practice2. The projection
can be regarded as a good work-around as long as correlations between input variables are negligible. Unfortunately this is almost never the case in real data (figure 3.6 showed a problematic example).
As shown in figure 5.4, naive Bayes builds 2·dim(input) histograms (distributions for “good” and “bad” in each input dimension). The histograms are filled with the training data and normalised to present a probability density with integral one. A decision for any evaluation position is made by calculating the product of the single probabilities for the “good” histograms on the one side and for the “bad” histograms on the other side. A continuous output in the range [0,1] is then given by
out= Q pgood,i Q pbad,i+Q pgood,i. (5.13)
In the case where no information is available (Q
pbad,i+Q
pgood,i= 0) out= 0.5 is taken.
p good,1 p p p good,2 bad,1 x1 x2 x1 x2 bad,2
good
bad
Figure 5.4: An example where the position (x1, x2) should be evaluated as “bad” because
pgood,1·pgood,2 < pbad,1·pbad,2.
Regression
The generalisation of this method to regression is problematic but possible. Instead of a pair of histograms for each input which represent the two classes we then need a large number of histograms for each input covering the range of the target value with a sufficiently fine granularity. If the range of the target value is binned into k intervals, the output is, for example, calculated as the target value in the bin with the maximum likelihood:
out=y[bin = arg maxk
Y
i
pki]. (5.14)
The granularity of the binning of the target value can be a serious problem as it demands a very high number of training examples if k grows large.
2
Four input variables with 20 bins each build a multidimensional histogram of 204
= 160000 bins each of which should be filled with a meaningful amount of data, say at least 10 in average. This requires 1.6 million training points at least.
5.4 Methods Based on Linear Separation 89