E.URB.MARIANO OSPINA RODRÍGUEZ 4 600 462 4 600 435 4 94 382 0.88 2 ESCUELA URBANA DE NIÑOS 8 352 27 8 352 320 8 450

To analyse the proposed retinal image classification mechanisms described later in this thesis a number of established classification techniques, taken from the domain of data mining, were used. Namely: Bayesian networks, SVM andk-NN. These algorithms are therefore described in some detail in the following three sub-sections.

3.6.1 Bayesian Networks

A Bayesian network is a probabilistic graphical model. It can be used to predict the probability that a given example belongs to a particular class, in this case the network is referred to as a Bayesian classifier. Bayesian classifiers are derived from Bayes theorem, thus ifT is a data tuple andH is a hypothesis that T belongs to classC, then:

P(H|T) = P(T|H)P(H)

P(T) (3.5)

where P(H|T) is the posterior probability of H given T (i.e. it is a measure of how confident we can be that H is true given that we knowT is true). Similarly,P(T|H) is the posterior probability ofT givenH. P(H) is the prior probability ofH andP(T)

is a prior probability of T. The most straight forward Bayesian classifier are founded on the Na¨ıve Bayes assumption [75, 83]:

1. Assume a training set withT tuples andm attributes, A1, A2, . . . , Am. Suppose also there arenclasses,C1, C2, . . . , Cn. Given a tuple,T, the classifier will classify

T to classCi if and only if:

P(Ci|T)> P(Cj|T) for 1≤j ≤n, j6=i (3.6)

whereP(Ci|T) is calculated using Bayes’ theorem as defined in equation (3.5),

P(Ci|T) =

P(T|Ci)P(Ci)

P(T) (3.7)

2. Based on equation (3.7), the probabilitiesP(A1, A2, . . . , Am|Ci) have to be com- puted in order to get P(T|Ci). Using Na¨ıve Bayes it is assumed that the at- tributes, A1, A2, . . . , Am, are independent of one another given any class label. Thus:

P(T|Ci) =P(A1|Ci)×P(A2|Ci)×. . .×P(Am|Ci) =Qm

s=1P(As|Ci)

(3.8)

3. A tuple,T, belongs to classCi if and only if:

P(T|Ci)P(Ci)> P(T|Cj)P(Cj) for 1≤j ≤n, j6=i (3.9)

P(T) as in equation (3.7) is omitted from the calculation as it is constant for all classes.

3.6.2 Support Vector Machines

A SVM is a classification system that tries to separate data of different classes by fitting ahyperplane(decision boundary), which maximise the “distance” between data representing two different classes provided the data is linearly separable [83]. Any new data may then be mapped onto the same space and classified according to which side of the hyperplane it falls. Given a dataset, D = {(X1,C1),(X2,C2), . . . ,(X|D|,C|D|)},

whereXis the training data and Cis the set of class labels with value{+1, -1}, linear SVM is constructed as follows [83]:

1. Find theoptimal separating hyperplane, known as the Maximum Marginal Hyper- plane (MMH) that maximally separates tuples of different classes in the space. Identification of MMH encompasses a number of steps:

a. Find separating hyperplane, which is defined as:

W·X+b= 0 (3.10)

whereWis the weight vector andbis a scalar value known as thebias,bmay be thought of as an additional weight w0. The hyperplanes that describe

each side of the separating “gap” are defined as:

H1 :W1X1+W2X2+. . .+W|D|X|D|+w0≥1 forCi = +1,

H2 :W1X1+W2X2+. . .+W|D|X|D|+w0≤1 forCi =−1. (3.11) where|D|is the number of tuples in the datasetD. Equation (3.11) shows that any data item that falls on or above H1 belongs to class +1, and any tuple that falls on or belowH2 belongs to class -1. Training tuples that fall onH1 orH2 are known assupport vectors. Equation (3.11) can be rewritten as:

Ci(W1X1+W2X2+. . .+W|D|X|D|+w0)−1≥0 ∀i. (3.12)

b. Find MMH. To obtained MMH, the problem is to minimise ||W||, subject to the constraint specified in equation (3.11). ||W|| is the Euclidean norm ofW. By minimising ||W||, which is equivalent to minimising 1₂||W||2_{, the}

distance between H1 and H2 will be maximised. This is achieved using an optimisation algorithm with Lagrangrian formulation and Karush-Kuhn- Tucker (KKT) conditions. Further details on how the MMH is derived can be found in [22]. Once identified, the MMH can be defined as a decision boundary: D(XX) = sv X i=1 CiαiXiX +b0 (3.13)

where sv is the total number of support vectors, Ci is the class label for the support vector (or training tuple) Xi, X is a test tuple, αi and b0 are

parameters determined by the optimisation algorithm.

2. Classify the test tuple. To achieve this, a test tuple, X, is applied to equation (3.13). If the sign of the computed results is positive,X is classified as +1. If the sign is negative,X belongs to the class −1.

The above process is used to train linear SVMs, where the training data is assumed to be linearly separable. This algorithm can be extended to learn nonlinearly separable training tuples by first transforming the nonlinear tuples into a higher dimensional space using a nonlinear kernel function. Three common nonlinear kernel functions are:

Polynomial: K(Xi,Xj) = (Xi·Xj+ 1)h

Radial basis function: K(Xi,Xj) =e−γ||Xi−Xj|| 2

Sigmoid: K(Xi,Xj) = tanh(kXi·Xj −δ)

Next, the constraint in equation (3.12) is rewritten to allow errors as follow [15]:

Ci(W1X1+W2X2+. . .+W|D|X|D|+w0)−1 +ξi≥0 ∀i. (3.14) where ξ ≥ 0 is called the slack variable that allow margin errors (the hyperplanes do not separate the training tuples of different classes correctly) and misclassification. To penalise the margin error and misclassification, subject to the constraint introduced in equation (3.14), a “soft parameter”C >0 is used to minimise 1₂||W||2 _{as follow:}

min1 2||W|| 2_+C L X i=1 ξi (3.15)

where L is the number of different classes. The optimisation algorithm to obtain the MMH and classification of test tuples as in case of the linear SVM can then be applied to classify test images.

3.6.3 k-Nearest Neighbours

A nearest neighbour classifier uses the most similar (closest) neighbouring object to label a new object [75, 83]. To identify the closest neighbouring objects, the distances between a new object and all other known objects are measured and the smallest se- lected. Distance can be measured by means of any distance metrics such as Manhattan or Euclidean distances. Generally,k-NN works in the following manner:

1. Assume a set of labelled training tuples,T, withm attributes (features),A1, A2, . . . , Am, that form a feature (or pattern) space with nclasses, C1, C2, . . . , Cn.

2. Given a test tuple,X, calculate the distance betweenX and all tuples in T. Dis- tance can be computed using any distance measures. If the Manhattan distance is used, the distance betweenX and a training tuple,Ti ∈T, is defined as:

d(X, Ti) = m X j=1 |AX_j −ATi j | (3.16) whereAX_j and ATi

j are values of attribute Aj of test tupleX and training tuple

3. Having computed the distance between X and all training tuples in T, given as

distance(X, T) ={d(X, T1), d(X, T2), . . . , d(X, T||T||)}, where ||T||is the number

of training tuples, sortdistance(X, T) in ascending order.

4. The test tuple,X, is classified based on itskclosest (neighbours) training tuples,

neighbours(X), using some voting mechanism. Note that the size ofneighbours(X) is defined by the value ofk. Thus, X is belongs to classCa if and only if:

4Ca>4Cb, 1≤a, b≤n, a6=b (3.17)

where4Ca denotes the occurrence of class Ca inneighbours(X).

3.6.4 Image Classification Applications

DM, or specifically image mining, has been applied widely to the classification of images. In the medical domain it has been applied to the diagnosis and grading of diseases, as well as categorisation. There are many examples. In [7] a statistical based classification approach was applied to identify grades of Anal Intraepithelial Neoplasia (AIN), a condition that may precede anal cancer. An approach for tumour classification using mammogram images was presented in [9]. In [152] image mining was applied to the classification of HEp-2 cells, cells that are used to identify autoantibodies that are useful to diagnose autoimmune related diseases. In [122] image mining was used to categorise various body parts within radiography images, while techniques to classify MRI brains scan images to identify human characteristics based on the shape of the corpus callosum were presented in [50].

Other domains where image mining for image classification has been applied include: remote sensing [128, 149, 186], the identification of flare state solar images [20, 175], the appearance and quality of potato chips [132] and others [8, 21, 144, 150]. In [128], various image classification techniques for remote sensing image classification were re- viewed and compared. For the classification of solar images, three techniques were applied, namely Multilayer Perceptron (MLP), Radial Basis Function (RBF) network and SVM [175]. In [132], the quality characterisation for classification of commercial potato chip images using Principal Component Analysis (PCA) and Linear Discrim- inant Analysis (LDA) was promoted. Neural networks were applied in [150] for the classification of images of various objects, whilek-NN was employed to classify images of various “scenes” in [8].