Los proyectos de ley sobre técnicas de reproducción artificial

For an algorithm to learn a concept or approximate a target function, it is necessary to observe a sufficiently long sequence of training examples generated from a stationary data distribution. This is the learning setup which is assumed in the experimental analysis presented in this section. We use all the available stationary datasets for regression tasks presented in Chapter 5. Since the real datasets used are relatively small, the evaluation was performed using the standard method of 10-fold cross-validation, using the same folds for all the algorithms included. For all the experiments, we set the input parameters of our algorithm to: δ = 0.01, τ = 0.05 and nmin= 200.

Due to the fact that most of the datasets are finite and of manageable size, an advantage is unintentionally given to the batch learners. Having a training set of finite size, batch learning algorithms are able to use more training examples in their early stages of building the tree, as long as the data can be loaded in main memory. They are also able to reuse the training examples at all levels of the tree. The incrementally learned trees, on the other hand, are expected to achieve similar performance as the batch learned ones, conditioned upon having a long enough stream of training instances. Therefore, all of the results presented should be analyzed, having in mind the aforementioned requirements for a successful relative comparison.

For a fair comparison, as well as to be able to evaluate the impact of linear models in the leaves, two separate evaluations were performed. First, FIMT without linear models in the leaves and without change detection (FIRT) was compared to two regression tree learning algorithms: CART, and M5’RT. Second, FIMT-DD with linear models included and without change detection (FIMT) was compared to six model tree learning algorithms: M5’MT, LR, CUBIST, BatchRD, BatchRA, OnlineRD, OnlineRA. We used two different versions of FIMT: FIMT Const with a constant learning rate, and FIMT Decay with a decaying learning rate.

The performance measures for CART, M5’ and FIRT averaged across the 10 datasets are given in Table 3. We have used the real-world datasets described in Chapter 5 Table 2. The presented performance results represent the average value for all the results obtained by using the 10-fold cross-validation. The detailed (per dataset) results are given in Table 29 in Appendix A section A.1.

84 Learning Model Trees from Time-Changing Data Streams

Table 3: Results from 10-fold cross-validation averaged over the real datasets for three regression tree learning algorithms. Relative error (RE), root relative mean squared error (RRSE), number of leaves, running time and the correlation coefficient (CC) are given, averaged across 10 datasets.

Algorithm RE % RRSE % Leaves Time (sec.) CC M5’RT 47.30 51.72 244.69 29.28 0.82

CART 75.63 62.77 4.18 9.67 0.75

FIRT 55.38 60.11 35.54 0.33 0.75

The comparison of regression tree learners shows that FIRT has similar accuracy (RRSE, CC) as batch learner CART and lower accuracy as compared to M5’RT. FIRT produces medium sized models (on average 35 leaves), which are larger than the CART trees (on average 4 leaves), but significantly smaller than the M5’RT trees (on average 245 leaves). The Nemenyi test confirmed that there is no statistically significant difference (in terms of the correlation coefficient) between FIRT and CART: this was also confirmed with a separate statistical analysis using the Wilcoxon signed-rank test.

Table 4: Results from 10-fold cross-validation averaged over the real datasets for model tree learning algorithms

Algorithm RE % RRSE % Leaves Time (sec.) CC M5’MT 42.10 46.09 76.71 29.54 0.85 LR 60.48 63.01 1.00 2.59 0.74 CUBIST 48.75 / 20.67 1.02 0.75 BatchRD 41.84 45.27 55.58 6.13 0.85 BatchRA 50.22 53.39 22.85 2.24 0.81 OnlineRD 49.77 53.26 9.98 32.67 0.80 OnlineRA 48.18 52.03 12.78 3.08 0.82 FIMT Const 60.21 104.01 35.54 0.42 0.70 FIMT Decay 59.40 74.11 35.54 0.42 0.73

The performance results of the model-tree learners averaged across the 10 datasets are given in Table 4, while detailed (per dataset) results can be found in Table 30 for the relative error (RE%), Table 31 for the size of the models (leaves) and Table 32 for the learning time (Appendix A section A.1). The comparison of model tree learners shows that FIMT Decay and FIMT Const have similar accuracy to LR, while the rest of the learners are better (BatchRD being the best). It is interesting to note that the split selection method of BatchRD which takes into account fitted linear models on both sides of each hypothetical split performs better compared to the selection method which is agnostic to the existence of linear models. Another interesting result is that the incremental learner OnlineRA achieves better accuracy than the commercial batch learner CUBIST. The analysis on the statistical significance of results (Friedman test for comparison of multiple classifiers followed by the Nemenyi post-hoc test) confirmed this, additionally showing that there is no significant difference between FIMT and OnlineRA, OnlineRD and BatchRA methods. There is a significant difference in accuracy between M5’, CUBIST and BatchRD over FIMT and LR, while no significant difference over BatchRA, OnlineRA and OnlineRD.

We have also noticed that FIMT Const exhibits high root relative squared error rates, which is a sign of the existence of predictions that deviate from the true value largely. The reason most likely is that some of the perceptrons have not been trained properly, due to a premature ending of the training or due to a divergence which may happen if the learning rate is set too high. The FIMT Decay version obviously avoids the divergence by reducing the step size as the tree improves its approximation of the target function. It was also observed that linear models in the leaves rarely improve the accuracy of the incremental regression

Learning Model Trees from Time-Changing Data Streams 85

Table 5: Results from holdout evaluation averaged over the artificial datasets Fried, Lexp, Losc with 10-fold cross-validation on a fold of 1M examples and a test set of 300k examples.

Algorithm RE % RRSE % Leaves Time (sec.) CC

FIRT 0.16 0.19 2452.23 24.75 0.98 CUBIST 0.13 / 37.50 104.79 0.98 FIMT Const 0.11 0.14 2452.23 27.11 0.98 FIMT Decay 0.11 0.14 2452.23 26.93 0.98 LR 0.46 0.51 1.00 2468.61 0.83 BatchRD 0.08 0.10 27286.30 5234.85 0.99 BatchRA 0.71 0.69 56.97 2316.03 0.51 OnlineRD 0.10 0.13 6579.5 3099.82 0.98 OnlineRA 0.70 0.68 57.77 2360.56 0.53

tree on these small datasets. This could be due to the fact that the regression surface is not smooth for these datasets and there is no need of using a model tree. The largest models on average are produced by M5’MT and BatchRD. In this comparison CUBIST is a special case, because it has a special mechanism for creating rules from the tree. Online learners have lower accuracy than their batch equivalents. FIRT has a significant advantage in speed over all of the learners. CUBIST is in the second place. Another advantage of FIRT is that it can learn with a limited amount of memory (as allowed by the user), while the other algorithms do not offer this option.

The artificial datasets (Fried, Lexp and Losc) were used to simulate a learning environ- ment in which enough training data are provided for the incremental learner FIMT. For a proper comparison, we proceeded again with a 10-fold-cross-validation procedure, for which we generated 10 random datasets, each of size 1 million examples, and one separate test set of size 300k examples. All the algorithms were trained and tested using identical folds. It should be noted that this type of evaluation is very difficult to perform in real-time for a re- alistic online learning scenario. The results in Table 3 show that the incremental algorithms are able to achieve better accuracy than the batch algorithms CUBIST, LR and BatchRA. OnlilneRD achieves better accuracy than CUBIST but, on the other hand, is significantly slower. Only the two versions of FIMT-DD were able to process the data faster than CU- BIST with a comparable accuracy. Detailed (per dataset) results can be found in Table 33 (Appendix A section A.1). The best algorithm is BatchRD whose models are much larger. BatchRA and OnlineRA give the worst results. The linear models in the leaves improve the accuracy of FIRT because we are now modeling smooth surfaces. The incremental model trees of FIMT Const and FIMT Decay have similar accuracy as OnlineRD, but are smaller in size and the learning time is at least 100 times smaller.