PROBLEMÁTICA-LIMITACIONES

The goal of feature selection is to select a subset of features that allow the classifier to reach optimal performance. Generally feature selection meth- ods can be divided into two groups: wrapper-based and filter-based feature selections. For wrapper-based techniques a classifier is involved in feature selection. The major disadvantages of wrapper-based methods are high com- putational cost and risk of over fitting. Whereas, in filter-based selection techniques, inherent characteristics of the data are used to select the feature. Their major disadvantage is to ignore the interaction between the features, as this technique selects each feature independently of the other.

Using feature selection for imbalanced data sets is a recent development with the majority of the work focused on text classification and web cate- gorization domains [47, 48]. Forman [47] investigated multiple filter-based feature ranking techniques.

Zheng at al. [49] observed that the existing measures used for feature selection are not appropriate for class imbalanced data sets. They proposed selecting features for positive and negative classes separately, and then explic- itly combining them. They used a filter-based technique to create one-sided and two sided metrics; one sided metrics select only positive features on their scores, and two sided metrics select both positive and negative features based on the absolute value of their scores. They then compared the performance of one sided and two sided metrics with different ratios of best positive and negative features. The ratio selection method results in better performance, compared to one and two sided metrics. Thus both positive and negative features are important to build a model. Castillo and Serrano [50] also used feature selection as the part of their studies. They used Genetic Algorithm [31] along with a filter-based feature selection method.

2.3.4

Ensemble Classifiers

An ensemble classifier uses a collection of base classifiers, instead of one sin- gle classifier, to make predictions. Breiman [51] used Bagging one of the ensemble methods, to reduce the prediction error by 20% on average over various problems. Boosting and Bagging are the most successful approaches of ensemble classification. The general ensemble approaches still have the same problem (low classification accuracy of minority class) in class imbal- ance applications but have been extensively used to handle class imbalance problems.

The AdaBoost (Adaptive Boosting) algorithm [29, 52] is an effective boosting algorithm to improve classification accuracies of any “weak” algo- rithm. It increases the weight of each example which has been misclassified by the preceding classifier. This forces the following classifier to focus on those examples which are more often misclassified. The AdaBoost algorithm is stated to be immune to over-fitting [53]. Therefore, Boosting is an at- tractive technique in dealing with class imbalance problems. In the 2-class problems some variants of the AdaBoost have been reported: AdaCost [37], Cost-sensitive Boosting (CSB1 and CSB2) [54], AdaC1, AdaC2 and AdaC3 [55]. These algorithms use cost information, also regarded as cost-sensitive boosting algorithms. As Boosting still suffer from low classification accuracy of minority class, the following variants are proposed for boosting algorithm: SMOTEBOOST [28] instead of increasing weights of any misclassified class creates synthetic examples for minority class; RareBoost [56] uses compari- son of True Positive (TP) and False Positive (FP) and True Negative (TN) and False Negative (FN) (TP, FP, TN and FN are explained in next section) in determining the weights of each misclassified example and RUSBoost [57] uses random under-sampling of majority class before boosting algorithm.

Chan and Stolfo [58] ran multiple experiments to find the ideal class distribution for building the training model and then used re-sampling to generate multiple training sets with the desired distribution. Each training set includes all the minority class examples and a subset of the majority class examples, where each majority class example is guaranteed to occur in

at least one training set, hence no information is lost. The classifier is applied on each data set and a meta-classifier is used to form a composite classifier from the resulting classification.

The same basic approach for partitioning the data into n subsets and learning multiple classifiers has been used in a study conducted by Yan et al. [59]. Yan et al. [59] investigate SVM (Support Vector Machine) ensembles built with partitioning and various techniques for combining n models to make the final classification decision. They propose hierarchical SVMs for aggregating the outputs of SVM ensembles.

Molinara et al. [60] investigate partitioning using two splitting methods: random splitting and clustering. Random splitting simply separates the ma- jority class into n independent disjoint random partition, then n models are constructed each using one partition of the majority class and the entire mi- nority class. For the clustering approach, they used k-means clustering to partition the majority class into k partitions. Each of the majority class par- titions are combined with the entire minority class to create k training data sets. Random under-sampling or over-sampling is used in each of the k train- ing sets to balance the class distribution. Their results showed that random splitting is superior to the cluster based approach. This is to be expected, since in k-clustering we can not control the number of majority class exam- ples in each cluster, and we may find a number of clusters where the majority class examples are less than the minority class. Moreover k-means is highly dependent on the choice of k, which is still an open question. Other cluster- ing approaches have been proposed to decompose complex decision boundary, e.g., non-linear separable boundary into linear separable boundary [61] and using clustering for under-sampling of the majority class [62, 63].

2.4

Evaluation Measures

Evaluation measures play a crucial role in both assessing the classification performance and guiding the classifier modeling. Traditionally, accuracy as define below is the most commonly used measure for these purposes. How- ever, for classification with the class imbalance problem, accuracy is no longer

a proper measure since the rare class has very little impact on accuracy as compared to the prevalent class [14]. For example, in a problem where a rare class is represented by only 1% of the training data, a simple strategy can be to predict the prevalent class label for every example. It can achieve a high accuracy of 99%. However, this measurement is meaningless in some applications where the learning concern is the identification of the rare cases.

When dealing with a binary classification problem we can always label one class as a positive and the other one as a negative class. The test set consists of P positive and N negative examples. A classifier assigns a class to each of them, but some of the assignments are wrong. To assess the classification results we count the number of True Positive (TP), True Negative (TN), False Positive (FP) (actually negative, but classified as positive) and False Negative (FN) (actually positive, but classified as negative). TP is also called ‘hit’, whereas FP is called ‘miss’ [64]. In biomedical applications, T P/P is

called ‘sensitivity’ whereas T N/N is called ‘specificity’ [65]. It holds that

T P +F N =P (2.2)

and

T N +F P =N (2.3)

The classifier has assigned T P + F P examples to the positive class and

T N +F N examples to the negative class. Let us define a few well-known and widely used measures:

specif icity = T N N ⇒1−specif icity = F P N =F P R (2.4) sensitivity = T P P =T P R=recall (2.5) Y rate = T P +F P P +N (2.6) precision= T P T P +F P (2.7)

accuracy= T P +T N

P +N (2.8)

missclassification error (MER) = 1−accuracy= F P +F N

P +N (2.9)

Precision, recall and accuracy (or MER) are often used to measure the classification quality of binary classifiers. The FPR (false positive rate) mea- sures the fraction of misclassified examples. The TPR (true positive rate) or recall measures the fraction of correctly classified examples. Precision mea- sures that fraction of positive examples classified that are true. Lift tells how much better a classifier predicts compared to a random selection. It compares the precision to the overall ratio of positive values (Yrate) in the test set. lif t= P recision P/(P +N) = Sensitivity Y rate (2.10)

2.4.1

Probabilistic Classifiers

Probabilistic classifiers assign a score or a probability to each example. A probabilistic classifier is a function f : X → [0,1] that maps each example

x to a real number f(x). Normally, a threshold t is selected for which the examples with f(x) ≥t are considered to be target example and the others are considered non-target examples.

This implies that each pair of a probabilistic classifier and threshold t

defines a binary classifier. Measures defined in the section above can therefore also be used for probabilistic classifiers, but they are always a function of the threshold t. Note that TP(t) and FP(t) are always monotonic descending functions. For a finite example set they are stepwise, not continuous. By varying t we get a family of binary classifiers.

Note that some classifiers return a score between 0 and 1 instead of a probability. For the sake of simplicity we shall call them also probabilistic classifiers, since an uncalibrated score function can be converted to a proba-

bility function.

2.4.2

F-measure

As mentioned earlier, accuracy is not suitable to evaluate imbalanced data sets, as accuracy places more weights on the majority class than on minority class, which makes it difficult for a classifier to perform well on the minority class. Due to this reason, additional metrics are coming into widespread use.

F-value (or F-measure) is a popular evaluation metric for imbalance prob- lems [66]. If only the performance of the positive class is considered, two measures are important: True Positive Rate (TPR) and Positive Predictive Value (PP value). In information retrieval, True Positive Rate is defined as recall denoting the percentage of retrieved objects that are relevant. Van Rijsbergen [67] defined the E-measure as a combination of Recall (R) and Precision (P) satisfying certain measurement theoretic properties:

E = 1−(1 +β 2_{)P R}

(β2)P +R (2.11)

F-measure also depends on the β factor, a parameter that takes a value between 0 and infinity and correspond to relative importance of precision vs recall. It can be shown that when β = 0 then F-measure reduces to precision and conversely when β → ∞ then F-measure approaches to recall. A β of 1 corresponds to equal weighting of recall and precision. To get a single measure of effectiveness where higher values correspond to better effectiveness, and where recall and precision are of equal importance, Lewis and Gale [68] define F_β=1 = 1−E_β=1.

F-measure (F ) is suggested to integrate Recall and Precision as an aver- age

F-measure = 2RP

R+P (2.12)

In principle, F-measure represents a harmonic mean between recall and precision

F-measure = 2

The harmonic mean of two numbers tends to be closer to the smaller of the two. Hence, a high F-measure value ensures that both recall and precision are reasonably high.

2.4.3

G-mean

When the performance of both classes is concerned, both True Positive Rate (TPR) and True Negative Rate (TNR) are expected to be high simultane- ously. Kubat et al. [26] suggested the G-mean defined as

G-mean = √T P R×T N R (2.14)