So far in this thesis we have concentrated on linear regression and how to per- form learning through the two mainstream approaches of SLT and Bayesian inference, highlighting the probabilistic benefits of the latter. We have seen how non-linearity between the input predictors and the responses can be achieved through basis function expansions while retaining the appealing nature and identifiability of linear models. In this section we take a step further into pos- sibly nonlinear embeddings of the original features and introduce the concept of kernel substitution, also known as the kernel trick, which has revolutionised
“Frequentists” on exactly the subjective nature of prior distributions. The interested reader is directed to (Jaynes 2003) and (Edwards 1992) for the Bayesian and Frequentist perspective respectively.
the field during the last decade (Sch¨olkopf and Smola 2002, Shawe-Taylor and Cristianini 2004).
Consider the basis function expansion of the linear regression model in Equa- tions 2.4 and 2.5 and generalise it to some possible nonlinear feature expansion Φ ∈ RN ×M with the ith row given by φ(x
i)
T
:
y = Φw + (2.21)
The likelihood and prior follow from Equations 2.10 and 2.15 after substi- tuting the expansion Φ. Disregarding the marginal likelihood which is a con- stant term we can express the logarithm of the posterior8 as the sum of the log-likelihood and log-prior:
log p(w|y, X, α, σ2) ∝ − 1 2σ2 N X i=1 {yi− w T φ(xi)}2− α 2w T w (2.22)
maximising with respect to w, setting to zero and solving for w we have:
w = − 1 ασ2 N X i=1 {yi− w T φ(xi)}φ(xi) = Φ T a (2.23)
where the vector a has elements ai = −ασ12{yi − w
T
φ(xi)}. We can now re-
formulate the logarithm of the posterior with respect to the parameter a and obtain: log p(w|y, X, α, σ2) = − 1 2σ2a T ΦΦTΦΦTa + 1 σ2aΦΦ T y − 1 2σ2y T y −α 2a T ΦΦTa (2.24) and we can see that the feature expansion Φ appears only as an inner product with itself. Hence this dual representation indicates that we actually only need inner products of the feature expansion and not the actual feature expansion per se. Defining the N × N Gram matrix K = ΦΦT as a symmetric matrix of vector inner products in an inner product space and setting the derivative to zero with respect to a we obtain the dual solution:
8We could directly formulate the closed form posterior as in 2.16 but we will maximise over
a = (K + ασ2I)−1y (2.25) which, recalling Equation 2.9, leads to the prediction for a novel sample x∗ as:
y∗ = w T φ(x∗) = a T Φφ(x∗) = k(x∗) T (K + ασ2I)−1y (2.26)
where k(x∗) denotes a vector of N inner products between the training set
expansion Φ and the test sample expansion φ(x∗).
This transformation of the problem leads to two main observations. First, that we do not need to explicitly construct a feature embedding Φ of the input samples but we only need to define a valid function that directly describes the inner product of some feature expansion. Secondly, the transformed regression parameters of the dual formulation are N dimensional now as they operate on the Gram matrix and they require an O (N3) inversion for estimation. This
appears initially disadvantageous as we were operating before on a space with dimensions equal to the number of basis functions, which are typically less than the number of samples, but it offers the advantage that implicitly now we can employ a very high (infinite in some cases) dimensional embedding.
Definition 2.1: [Kernel function] (Shawe-Taylor and Cristianini 2004) A kernel is a function k that for all xi, xj ∈ X satisfies
k(xi, xj) = hφ(xi), φ(xj)i
where φ is a mapping from X to an (inner product) feature space F φ : x 7→ φ(x) ∈ F.
From Definition 2.1 we can see that the Gram matrix K is the corresponding kernel matrix and now we can employ any valid kernel function to implicitly produce high dimensional embeddings. The main kernel property of interest at this stage (see (Shawe-Taylor and Cristianini 2004) for a full treatment) is that the resulting kernels are symmetric positive semi-definite matrices.
Definition 2.2: [Positive Semi-definite Matrix]
Some typical kernel functions k (xi, xj) = hφ(xi), φ(xj)i that are employed
in this thesis are summarised in Table 2.3:
Kernel Type Function Characteristics
Linear (Cosine) xTixj
Cosine follows by normalisation Polynomial xTixj + 1 n Degree n Gaussian (RBF) exp −||xi− xj|| 2 2σ2
Infinite degree polynomial
Finally, revisiting the linear regression setting we can now reformulate it into a kernel regression problem:
y = wTK + (2.27)
and as before we obtain a closed form Gaussian posterior distribution for the regression coefficients as p(w|y, K, α, σ2) = N (µ, Σ) with parameters defined
with respect to the kernel matrix K as:
µ =KTK + σ2αI
−1
KTy (2.28)
Σ = σ2(KTK + σ2αI)−1 (2.29)
The kernel trick offers a powerful and efficient way of producing high di- mensional data embeddings that capture non-linearities of the modelling phe- nomenon and will be of especial interest to the classification setting that we visit next.