Antecedentes Teóricos

Figure 5.9 shows a critical difference diagram for NB, SSNB, SSNB-sigmoid, SSNB-sigmoid- fb, SSNB-sigmoid-LODO, SSNB-sigmoid-cv. Clearly, the logistic transformation improves the SSNB classifier. If the biased protocols are used with the logistic transformation trick, SSNB-sigmoid and SSNB-sigmoid-fb, then the result obtained is better than SSNB and NB, especially if the average value of the q_i variables was fixed. In addition, if the unbiased protocols are used with logistic transformation, SSNB-sigmoid-LODO and SSNB-sigmoid- cv, then the result obtained is equivalent to the NB classification performance but at least superior to the SSNB classifier. This result suggests that using the logistic transformation in the E-step of the EM based semi-supervised learning has not significantly improved the performance of NB classifier but it is a step that at least guides the algorithm better than just

the SSNB classifiers. The logistic transformation is likely to be beneficial, but the problem with estimating the parametersα andθ using the labelled data is unreliable.

CD 6 5 4 3 2 1 2.1071 _{SSNB-sigmoid-fb} 2.4821 _SSNB-sigmoid 2.8571 _NB 4.1964 SSNB-sigmoid-LODO 4.4286 SSNB-sigmoid-cv 4.9286 SSNB

Fig. 5.9 Critical difference diagram for the NB, SSNB, SSNB-sigmoid, SSNB-sigmoid-fb, SSNB-sigmoid-LODO and SSNB-sigmoid-cv over 12 discrete and 16 continues benchmark datasets. It shows that there are statistically significant differences between the means ranks for the SSNB, SSNB-sigmoid-LODO and SSNB-sigmoid-cv classifiers and NB, SSNB- sigmoid and SSNB-sigmoid-fb classifiers, however there are no statistically significant differences between the mean ranks for the NB, SSNB-sigmoid and SSNB-sigmoid-fb classifiers which are linked by the bar

5.5

Conclusions

In this chapter we started off by describing problem in the NB classifier, in which the predicted class probability were overconfident, being extremely close to 0 and 1. Then, we showed the effect of this problem in the NB classifier to the EM algorithm base semi- supervised learning when it used both labelled and unlabelled data to estimate the model parameters. Consequently, the main research question was established: could spreading out the predicted class probabilities for the unlabelled data improve the NB classifier? In order to do that, the logistic transformation method was used with various approaches to choose the logistic function parameters was used in a series of experiments. In the first experiment, the predicted class probabilities for unlabelled data were spread out and we as- sumed that the correct average value of the labels for unlabelled data was known by using the SSNB-sigmoid-fbapproach. This approach can improve the baseline classifier because the SSNB-sigmoid-fb has a very biased protocol. Next, theSSNB-sigmoidapproach was used

without assuming that the correct average value of the labels for unlabelled data was known, which has a slightly biased protocol. The experiment results show that the SSNB-sigmoid approach is better than the NB classifier but there is no statistically significant difference. The transfer learning method was used with the SSNB-sigmoid approach in another exper- iment known asSSNB-sigmoid-LODO. The goal behind using transfer learning is to use a default value for the rate of spread out of the predicted class probabilities for unlabelled data. Finally, the cross-validation method,SSNB-sigmoid-cv, was used for choosing the logistic transformation function parameters. The rank of the NB classifier is higher than the SSNB-sigmoid-LODOandSSNB-sigmoid-cv,but there is no statistically significant differ- ence. The SSNB-sigmoid approaches, (SSNB-sigmoid-fbandSSNB-sigmoid), which are biased protocols, do help a bit and are approximately equivalent to the NB classifier, while in both casesSSNB-sigmoid-LODOandSSNB-sigmoid-cv, logistic transformation methods do not help because both are unbiased protocols. In this chapter, we conclude that less improvement in the NB classification performance can be obtained by using less unbiased protocol. The logistic transformation method improve SSNB, but not to the point where it is a significant improvement on NB. The logistic transformation method just moderates a little bit but the experiments results suggest that does not help very much. However, it does generate better performance than the SSNB classifier.

Investigation of active learning

In previous chapters, various experiments were performed and discussed in order to evaluate whether semi-supervised learning can improve the performance of the naïve Bayes classifier. In this chapter, we present results from active learning experiments for selectively labelling large amounts of unlabelled patterns. The dual goals of using active learning methods are the creation of high-quality labelled data to improve classification performance, and the minimisation of the manual labelling effort required. This chapter starts with a reproduction of some existing experimental results and a description of the experimental design for the real world benchmark datasets. It continues with a discussion of results obtained using the suite of benchmark datasets, including the evaluation measures and statistical tests used. After selecting the least confidently unlabelled pattern by active learning especially in the early stages, we attempt to use the remaining the large amount of unlabelled patterns via semi-supervised leaning to increase the classification performance. Thus, we design some experiments to combine active learning with semi-supervised learning.

6.1

Introduction to active learning

A limitation of supervised learning methods is that they require the training data to be labelled, which is often a time consuming and expensive task. Semi-supervised techniques were used previously to extract predictive information from unlabelled patterns. Nevertheless using unlabelled patterns does not generally improve classification performance in the case of a naïve Bayes classifier. In this chapter, we investigate active learning methods that acquire labelled data incrementally, using the existing model to select particularly helpful additional training patterns for labelling by an oracle. Therefore, active learning methods may reduce the number of examples that must be labelled to achieve a particular level of accuracy. Achieving the same naïve Bayes classification performance with fewer labelled training examples is an important step towards improving the classifier and previous works have shown that this can be done successfully [52].

As mentioned in Chapter 2.3, active learning is a machine learning framework that allows the classifier to automatically select the most informative unlabelled patterns for manual labelling. Thus, the main goal in active learning can be referred to as selective sampling. Different sampling strategies exist, for example, uncertainty sampling is a specific type of active learning which selects informative patterns. These patterns are near of the decision boundaries and so have the lowest confidence score. In this section, we are interested in implementing uncertainty sampling for pool-based active learning, which provides a small set of labelled data,L, and a large set of unlabelled data,U, for training. With the small amount of labelled patterns the initial parameters of model are estimated, this method has been applied by Ramirez-Loaiza et al. [64]. Therefore, before investigating the main hypothesis, we first attempt to reproduce the experiments from Ramirez-Loaiza et al. [64] as a preliminary investigation.

The preliminary experiment attempted to reproduce the ALNB (Active Learning Naïve Bays) classifier results with theHivabenchmark dataset that was originally reported in Ramirez-

Loaiza et al. [64] which is a recent empirical evaluation paper that used ten well-studied benchmark datasets to demonstrate the effectiveness of the ALNB classifier. The Hiva benchmark dataset is one of the datasets used, which is a real world binary classification problem with 42,678 examples and 1,617 features, that was originally developed for an active learning challenge by Guyon et al. [34].

The authors of [64] designed their experiments as follows: the performance of the ALNB classifier was averaged over five runs of five-fold cross validation. For each experiment, ten labelled patterns (five from each class ) are chosen from the available data and the rest of the training set was treated as the unlabelled pool,U. However, if the unlabelled pool,U, consisted of more than 10,000 patterns, then 10,000 patterns were randomly sub-sampled from the large unlabelled pool. For computational convenience, at each iteration the top ten most informative instances were selected, as determined by the AL strategy. Following this experimental design, we obtained the average recall learning curve plot forHivabenchmark as shown in Figure 6.1.

200 400 600 800 1000 1200 1400 1600 1800 2000 2200

Number of labels queried

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Recall hiva AL UCS NB

Fig. 6.1 Average test set recall learning curve for the Hiva benchmark dataset using the uncertainty sampling strategy of ALNB classifier over 5-run 5-fold cross validation

The learning curve obtained for the ALNB classifier suggested that current experimental results were approximately the same as the learning curve fromHivabenchmark dataset given by, [64] suggesting our implementation of ALNB is correct.

After this experiment, we also reproduced the results described in Antonucci et al. [4], which evaluated the uncertainty sampling strategy with pool-based active learning for datasets with various continuous features. This paper is an empirical evaluation for AL algorithms for classification tasks, one of the classifiers investigated being NB, on different data sets from the UCI repository. In order to reproduce some of the results from Antonucci et al. [4], two benchmark datasets,diabetesandiris, were chosen from UCI repository. Thediabetes benchmark dataset is a binary classification problem, which consist of 768 examples and 8 features, whereas theirisdataset has three classes with 150 examples and 4 features. The author implemented uncertainty sampling as follows: 50 replications were generated for each dataset and the average of the accuracy learning curve was calculated on the test data. The datasets are partitioned to form the training and test sets in a stratified way. Antonucci et al. [4] randomly draw 10 patterns as the labelled training set and 100 patterns as a test set, then the remaining patterns are treated as unlabelled training patterns. The procedure was started by training on ten labelled patterns,L=10, to estimate the initial parameters for the initial NB classifier. Then, the predicted probabilities of the labels for unlabelled patterns were ranked according to uncertainty sampling. five patterns with the highest uncertainty score were then selected and submitted for labelling by the oracle. Then, newly labelled patterns were added to the labelled training patterns and removed from the large unlabelled pool. Finally, the labelled training examples were used to update the parameters of a standard NB classifier. Then the accuracy of ALNB classifier is evaluated on the test set. This procedure is repeated until the active learning set is empty.

Figure 6.2a shows the mean accuracy of the ALNB classifier for thediabetesbenchmark using uncertainty sampling with pool based active learning. Figure 6.2b shows the same

algorithm for the irisbenchmark dataset. The results obtained suggest that the current learning curves for bothdiabetesand iris are equivalent to the learning curves of the Antonucci et al. [4]. Based on the initial experiments results, we then evaluated the ALNB classifier over a larger suite of benchmark datasets.

100 200 300 400 500 600 700

Labeled set size 0.55 0.6 0.65 0.7 0.75 0.8 Accuracy diabetes dataset ALNB 5 10 15 20 25 30 35 40 45 50 55

Labeled set size

0.7 0.75 0.8 0.85 0.9 0.95 1 Accuracy iris dataset ALNB

Fig. 6.2 The average accuracy of the ALNB classifier for (a)diabetesand (b)iris