Presentación de resultados.

unbalanced data. The AUC (of the evolved classifiers) and GP training times of the new fitness functions are compared to other approaches from the previous chapter, and two other machine learning algorithms (Naive Bayes and Support Vector Machines) on the tasks.

4.1.1

Chapter goals

This chapter has two mains goals. The first goal is to develop new measures in the fitness function to find solutions with high AUC on these tasks. These new fitness functions aim to improve AUC performances over the traditionalAve, and improve training times over the traditional Auc. The second goal investigates whether an equal weighting of the minority and majority class accuracy in a weighted-average fitness function, or a non-equal weighting where one class has a greater cost in fitness than the other, finds solutions with better overall AUC on the tasks.

4.2

Current Approaches in Fitness

This section summarises the GP framework defined in the previous chapter (also used in this chapter), and presents several baseline fitness functions in GP to evaluate the effectiveness of the new measures (developed in the next section). These baseline fitness functions include the three approaches from the previous chapter,Acc, Aveand Auc, and four other useful approaches from the literature, namely,W ave,AveM,AucE andW mw.

4.2.1

GP Framework

The same GP framework is used in these experiments as outlined in the previous chapter. To recap, the genetic program solutions use a tree-based structure for representation. Feature and constant terminals are used in the terminal set, and the function set consists of the four standard arithmetic operators (+,₋,% and

×) and a conditional operator (if). As the previous chapter establishes that the standard zero-threshold (ZT) strategy performs as well as the non-static strategy on these tasks, the ZT strategy is used to translate the real-valued output of a solution (when evaluated on an input instance) to the target class labels. Recall that this strategy uses the natural division between positive and

Table 4.1: Outcomes of a two-class classification problem.

PredictedPositive Class PredictedNegative Class Actual Positive Class True Positive (TP) False Negative (FN) Actual Negative Class False Positive (FP) True Negative (TN)

negative numbers to represent the two class labels, i.e., an input instance will be assigned to the majority class if the solution output is negative, otherwise it will be assigned to the minority class.

4.2.2

Baseline GP Fitness Functions

This chapter presents several baseline fitness functions in GP. These include the three measures discussed in the previous chapter (Acc, Ave and Auc), two average-based measures (AveM and W ave), and two alternative approaches to calculate the AUC is fitness (AucE andW mw).

Fitness FunctionsAcc,AveandAuc

The three fitness functions from the previous chapter are repeated below for convenience. These include the standard GP fitness function Acc (the overall classification accuracy), and average accuracy of the minority and majority class

Ave, and the AUC in the fitness functionAU C. These fitness functions are defined using the four outcomes for binary classification shown in Table 4.1. Note that the same confusion matrix is also shown in the earlier chapters (Chapters 2 and 3) but is repeated here for convenience.

Acc= T P+T N T P+T N+F P+F N (4.1) Ave = 1₂ T P T P+F N + T N T N+F P (4.2) Auc=PN−1 i=1 1 2(F Pi+1−F Pi) (T Pi+1+T Pi) (4.3)

Average-Based Fitness FunctionW ave

An alternative form of Ave uses the weighted-average of the minority and majority class accuracy in the fitness function, as shown by Eq. (4.4). In W ave, the weighting coefficient controls the trade-off between the minority and majority

4.2. CURRENT APPROACHES IN FITNESS 79 Table 4.2: Minority and majority class accuracies of three solutions, and the correspondingAveM fitness values.

Solution Minority Accuracy Majority Accuracy AveM Fitness

a1 70% 70% 24.5%

a2 60% 80% 24.0%

a3 85% 55% 23.4%

class accuracy where0 < W < 1. WhenWis 0.5, the accuracy of both classes is considered as equally important in fitness (with this setting, W aveand Ave are the same). When W > 0.5, minority class accuracy will contribute more in the fitness function than majority class accuracy by factorW. Similarly, majority class accuracy will contribute more whenW <0.5.

W ave=W _{× T P}T P_{+F N}+ (1₋W)_× T N T N+F P

(4.4) This fitness function addresses the second goal of this chapter, and investi- gates whether an equal weighting of the minority and majority class accuracy (W = 0.5), or a non-equal weighting (W ₆= 0.5), finds solutions with better overall AUC on the tasks.

Average-Based Fitness FunctionAveM

The fitness function AveM or Eq. (4.5) is based on the geometric mean of the minority and majority class [57][141]. Similar to Ave and W ave, the two components in Eq. (4.5) corresponds to the minority and majority class accuracy, respectively. AveM = 1 2 T P T P+F N × T N T N+F P (4.5) This fitness function has the two useful properties compared to the arithmetic average (used inAve and W ave). Firstly, as the geometric mean multiplies the minority and majority class accuracies, if the accuracy on a single class is zero, then zero is returned. Secondly, AveM can produce more fine-grained fitness values when Ave evaluates to the same fitness value for some combinations of the two components.

For example, consider the minority and majority class accuracies of three solutions (a1, a2 and a3) shown in Table 4.2. Each solution has the same average

accuracy, 70%, and therefore the same fitness values according toAve. In contrast,

shown in Table 4.2. This fitness function aims to investigate how these differences affect the AUC of evolved solutions.

AUC-Based Fitness FunctionsAucF andAucE

As discussed, a major limitation of usingAucor Eq. (4.3) in the fitness function is the increased training times, due to the computational effort required to construct an ROC curve in fitness evaluation. Each classifier must be evaluated on all fitness cases N times to obtain N distinct TP/FP points on an ROC curve. A useful technique to speed-up training times forAucuses fewer TP/FP points on the ROC curves [84]. In [84], exactly seven distinct TP/FP points are used in the ROC curves; this number is recommended for a fast and accurate approximation to the full AUC [84]. To simulate this in GP, seven ROC points are generated by choosing seven distinct class thresholds spread uniformly over the range of a given genetic program’s output values (when evaluated on all training instances). The genetic program is then evaluated at each threshold to produce the seven TP/FP points. Naturally, this faster approximation of the AUC will have a lower precision than the full AUC. Recall (from the previous chapter) that in the full AUC, every distinct value in the range of output values for a given genetic program is taken as a separate class threshold.

This chapter compares the GP training times and AUC of the evolved solutions using both AUC calculations in the fitness function, i.e., the full AUC

AucF, and and the faster estimationAucE (which uses exactly seven ROC points).

AUC-Based Fitness FunctionW mw

An alternative technique to calculate the AUC in the fitness function uses a statistical approximation based on the Wilcoxon-Mann-Whitney (WMW) statistic [84][179][57], as shown in Eq. (4.6). The WMW statistic uses a series of pairwise comparisons between the genetic program outputs (when evaluated on examples from the two classes), effectively measuring the ordering of minority to majority class outputs. In Eq. (4.6), Pi and Pj represent the output of a genetic program when evaluated on an example from the minority and majority class, respectively, andNminand Nmaj are the number of examples in the the minority and majority class respectively.

The indicator functionIwmwenforces two constraints on the ordering of output values for each class. The first constraint (Pi ≥ 0) checks whether the minority class outputs are zero or positive. The second constraint (Pi > Pj) checks whether

4.3. NEW FITNESS FUNCTIONS 81