Diseño metodológico

for the VC-Dimensions of certain learning algorithms. A tight upper bound for the VC- Dimension can be calculated by using the margin of any hypothesis that is put out by the learning algorithm. The margin is the least distance of training events to the decision boundary in the separable case (compare the discussion of the support vector algorithm in section 5.4.3).

4.5

Criticism: No-Free-Lunch Theorems

Wolpert [46] uses an error quantity that is different from the IID true error EP as defined above. He wants to emphasise the importance of the generalisation property of learning algorithms and thus restricts the true error to all points of the input space X which are not in the training set. Theoff-training-set error is therefore

EO(h) =

X

x∈X,x6∈T

L³h(x), y(x)´Px(x). (4.15)

This definition makes again clear that the theoretical framework is commonly based on a finite or at least countable input space X, compare the footnote on page 72.

Since the PAC and VC frameworks provide results about the relation between training error ET and IID true error EP the question arises whether also for off-training-set error one could provide similar bounds.

No-Free-Lunch Theorem (NFL)

For any two learning algorithms A and B the mean off-training-set errors av- eraged over all target functions y are exactly the same for any training set T:

EO(hA) =EO(hB).

It is important that the off-training-set errors are averaged over all target functions y. Since one usually has no a priori information about the target function it seems reasonable to average over all of them. Intuitively this theorem tells us that the target function may – theoretically – have any value for all the points which have not been in the training set. This means further that any guess for these values is as good as random guessing if the resulting errors are averaged over all target functions. An extreme example is shown in figure 4.2.

Concerning strategies like “low training error and low VC-dimension gives high prob- ability for low true error” or even concerning the simple strategies presented in sec- tion 3.12 like Cross-Validation or Bootstrapping, the No-Free-Lunch theorem states that these strategies work in as many cases as they fail. The No-Free-Lunch theorem makes every attempt to have low true error seem useless. To clarify the apparent contradiction of this theorem and the frameworks presented above we will discuss one particular aspect for each framework:

• The Bayesian formalism makes use of prior probabilities. The optimality of the Bayesian approach seems to be questioned by the NFL theorem because its algorithm to select the optimal hypothesis seems to be just a learning method to which the NFL theorem applies. But the NFL theorem does not apply because no averaging over target functions would make sense since the Bayesian formalism makes an assumption

78 4. Statistical Learning Theory

Figure 4.2: The No-Free-Lunch theorem makes local density estimation seem useless: In this example the point marked by a cross could be an open or a filled circle with equal probability. Despite the fact that this position is surrounded by filled circles so that it seems obvious that the cross is also a filled circle, both possible target functions have to be taken into account with equal probability since no prior information is given. Such a prior for local density estimators would favour locally constant target functions.

about the prior probabilities of the target functions as the first step. This makes clear that any assumption about the prior probabilities of the target function forbids the application of the NFL theorem. Such kind of priors in the form of favour for “simple” functions as discussed below in section 4.6.1 are criticised by Wolpert to be not justified from the probabilistic point of view.

• PAC-Learning (and also the VC-framework) uses IID error and not off-training- set error. The results that connect the training error with the true error in the PAC framework rely on the IID definition of the true error. In contrast, for the off- training-set definition of the true error the NFL theorems apply which means that the value of the training error and the size of the hypotheses space H cannot give any hint about the off-training-set error. Again it is important to note that the PAC framework works with a countable input space which ensures that the rising number of training eventsnmakes the input space more and more known in terms of covered probability density Px. Therefore the bound on IID error is naturally connected to the number of training events. For off-training-set error of course there is no such connection.

• Understanding the conditional probabilities in the VC framework is important. This understanding is not directly related to the NFL theorem. Nevertheless it be- longs to the general criticism of the VC-Framework as it questions the most common interpretation of the resulting theorem. This interpretation assumes a high proba- bility guaranteed that, given a certain value for the training error, the true (IID) error is not too far away. However, Wolpert emphasises that the correct conditional probabilities imply the following statement (and only this one): Given a fixed value of the true (IID) error there is a high probability that the training error is not too far away. This situation does not reflect the application in real life where a training error is measured and the true error should be estimated. For a conversion of the conditioned probability P(ET|EP) into the needed P(EP|ET) one would use Bayes theorem. But then the unknown probabilities P(EP) (which in fact matter) would be needed (compare the discussion in section 3.13.1).