Adaptación de los campos en su disputa

The results described so far in this chapter were obtained by parameterising the neural networks with the suggestion made in the previous chapter (pmeta_{) and using} the transferred weights. We now analyse the impact of transfer for neural networks parameterised with the best parameterisation found in the grid search (pgrid_{) aiming} at studying the impact of transfer learning in the case where the parameterisation suggested is closer to the one obtained by grid-search.

First, we analyse the proportion of positive transfers achieved with SCor-mapped transfers. Figure 5.7 shows the comparison of the proportion of positive transfers obtained when considering SCor mapping method.

0 10 20 30 40 (a) MSE0 −10 0 10 20 (b) duration −20 −10 0 10 20 30 (c) MSE

Figure 5.7: Percentage of positive transfers obtained with SCor-mapped transfer (relative to the random transfer). The xx axis represents the target datasets on the same order as in Table A.1 in Appendix A. The yy axis, for each performance metric, corresponds to the proportion of positive transfers (relative to the proportion of positive transfer obtained by random transfer) when considering SCor mapping method.

Similarly to Figure 5.4, positive values mean that SCor-mapped transfer leads to positive transfers more often than random transfer. We observe that SCor-mapped

transfer generally leads to more positive transfers than random transfer, except for five datasets for the duration metric and four for the MSE metric. These are presented on Table 5.7, where we can see that the differences are not too large.

Table 5.7: Datasets in which random transfer shows larger proportion of positive transfers for each evaluation metric considered.

(a) duration dT R SCor 5 7 47 41 5 12 56 44 5 14 26 21 5 18 41 38 11 3 0 (b) MSE dT R SCor 2 2 71 56 5 7 74 56 16 79 65 17 56 50

However, as stated before, the proportion of positive transfers is not enough to assess the best transfer results. Figure 5.8 shows the impact obtained by random transfer and SCor-mapped transfer for each performance metric in each dataset.

R SCor MSE0 dur ation MSE −100 −50 0 50 100 −100 −50 0 50 100 −100 −50 0 50 100

Figure 5.8: Impact of the transfers for each metric with random transfer and SCor- mapped transfer. The xx axis represents the target datasets on the same order as in Table A.1in Appendix A. The yy axis represents the highest impact obtained for each target dataset.

As we can see in the figure, there is only one case in which SCor-mapped transfer leads to negative transfer, while random mapping leads to several negative transfers.

5.5. SUMMARY 85

Furthermore, we present the best transfer found for each target dataset with the SCor- mapped transfer for both parameterisations considered: pmeta_{(TableH.1, AppendixH)} and pgrid _{(Table I.1, Appendix I).}

5.5

Summary

In this chapter we studied the impact of transfer learning on neural networks for regression problems. The experiments were performed with a set of 28 datasets and we tried transfer learning for every possible combination of source/target datasets (the source is always different from the target). The transfers were performed considering two parameterisations: the one suggested by the metalearning method described in the previous chapter; and the best parameterisation found in a grid search. The objective of using this last one is to assess the transfer learning results if the metalearning model suggested a parameterisation that is closer to the best possible (from within the ones tested).

Results indicate that, provided that the source dataset is well chosen, transfer learning can be used to initialise neural networks in order to increase their computational performance, while not harming (and, sometimes, even increasing) their predictive performance.

Also, this is not due to the initial weights having a distribution closer to the optimal. If it was the case, random transfer would lead to similar impacts as mapped transfer and this is not the case. Mapped transfers lead to higher impacts and SCor-mapped transfer revealed to achieve better results and so it was chosen to be used for the rest of the work. In the following chapter we study how metalearning can be used to select a good source dataset for a new target dataset.

Chapter 6

Metalearning for source selection in

heterogeneous transfer learning for

neural networks

In this chapter we describe the study performed to answer RQ4 (Chapter 1): Can

metalearning be used to support transfer learning in neural networks? Our objective is to use metalearning to predict if transferring weights from a specific source network will make the target network converge faster, without harming its performance. This will be performed according to the transfers’ characteristics (metafeatures).

We propose seven sets of metafeatures for the selection of the source network for a specific target network (Section 6.1). These metafeatures aim at capturing the transfers’ characteristics that can be used to decide whether a specific transfer will be advantageous for a determined target dataset. The metafeatures are then used by the metalearning that tries to map the data characteristics to the impact of a certain transfer.

We perform an extensive experimental setup (Section6.2) to validate our method and its results are presented in Section6.3). We present two resources developed that allow data scientists to fully configure neural networks for the R package nnet (Section6.4) and finish with a summary of our observations (Section 6.5).

6.1

Metafeatures for source network selection

The purpose of the metafeatures is to characterise the transfers. These characteristics are then used by the metalearning and mapped to the impact of the transfers on the neural networks.

We propose metafeatures for the task of selecting a neural network to be used as source for initialising a specific target network. Our objective is that, by initialising the target network with weights coming from a previously learned source network, the target network’s performance will be improved.

We start by generating three separate groups of metafeatures specific for characterising the transfers (described next in Subsections6.1.1, 6.1.2and 6.1.3) and then aggregate the groups of metafeatures in seven sets (see Subsection 6.1.4).