Se señala el plazo de tres meses para la duración de cada

Prior to training the multi-label models, the modelling conditions for each label (transporter) were optimized. From a set of 5 possible feature selection techniques, and 3 possible classifiers (with and without misclassification cost), the best combination of features, classifier and cost was optimized for each label, based on the highest internal validation performance (best G-mean). Table 5.2 summarizes the optimal conditions achieved for each label.

As shown in Scheme 5.1, a total of 1951 multi-label models were produced (1950 CC models + 1 BR model). From these, the best model was selected based on the highest average bACC on the validation set. In this CC model, the order of labels was OCT1, OATP2B1, OATP1A2, PEPT1, OATP1B1 and finally OATP1B3. Its predictive performance on the test set, at a single-label level, is summarized in Table 5.2. The results can be compared with Table 5.3, where the test performance of the corresponding single-label

models from the BR multi-label model (non-label-interaction equivalent of CC) is summarised.

Table 5.2. Single-label test set performance of the best CC model. Predicted transporter binding

labels used as features are generically presented with the respective transporter prefixed by the letter

“p”.

Label _classifierBest

Best feature selection Best misclassification cost Sen (%) Spe (%) G-mean (%) Labels present as features in the models

#1: OCT1 C4.5 C4.5-GA Equal cost 90.9 64.7 76.7

Not applicable; first label in chain

#2: OATP2B1 C4.5 RF-GS 1.3*FN 50.0 85.7 65.5 pOCT1

#3: OATP1A2 BT GS Equal cost 90.9 25.0 47.7 pOCT1, pOATP2B1 #4: PEPT1 C4.5 (UOBag) GS 2*FP 81.3 57.1 68.1 pOCT1, pOATP2B1, pOATP1A2 #5: OATP1B1 RF C4.5-GA 3*FP 68.4 83.3 75.5 pOCT1, pOATP2B1, pOATP1A2, pPEPT1

#6: OATP1B3 _(SMOTE)RF GA Equal cost 92.3 66.7 78.4

pOCT1, pOATP2B1, pOATP1A2, pPEPT1, pOATP1B1

Table 5.3. Single-label test set performance for the BR model equivalent to the best CC model.

Label Sen Spe G-mean

OCT1 90.9 64.7 76.7 OATP2B1 50.0 100.0* 70.7 OATP1A2 90.9 25.0 47.7 PEPT1 68.8 57.1 62.7 OATP1B1 68.4 83.3 75.5 OATP1B3 84.6 66.7 75.1

At a multi-label level (i.e. looking at the set of classifiers), the CC model showed improved performance compared to the BR model (Table 5.4), across almost all performance measures, with the exception of Spe-L. However, this can be attributed to an increase in Spe from 85.7% to 100% for the OATP2B1 label within the BR model, as the remaining

single label models have the same Spe values in Tables 5.2 and 5.3. Even though both HL and F1 are not statistically different between models (p > 0.05), this was expected given that both models output a very similar set of predictions, which does not allow for a differentiation at a global scale.

Table 5.4. Multi-label performance obtained on the test set. Recall that HL is to be minimized, whilst

the other measures are to be maximized. Statistical testing was only carried out for the instance- based measures, as explained in the Methods section 5.2.

measure BR CC p-value Label- based bAcc 68.1 68.7 n.a. Sen-L 75.6 79.0 n.a. Spe-L 66.1* 63.8 n.a. Instance- based HL 26.4 22.8 p = 0.164 F1 70.7 71.9 p = 0.128 (S); p= 0.671 (NS)

n.a. – non applicable. *results from single-label performance measure marked in Table 5.3 (see text).

In order to understand the factors that led to the superior multi-label performance of the CC model, both models were compared at the single-label level. Tables 5.2 and 5.3 show that three out of the six labels in the multi-label CC model show no improvement over the standalone single-label models; on the other hand, PEPT1 and OATP1B3 show improvement in the CC model, which results from a marked increase in Sen accompanied by maintained Spe. The classifiers for both PEPT1-CC and OATP1B3-CC selected all available previous labels as predictors, and considering that the introduction of previous label predictions was the only differing aspect between the BR and the CC models, it can be concluded that this was the driving factor of the improved performance of PEPT1-CC and OATP1B3-CC. Moreover, it should be highlighted that all previous label predictions used in the modelling of PEPT1 refer to transporters that seemingly have no obvious structural (Schlessinger et al., 2013b) or physiological link (Cesar-Razquin et al., 2015) to it (Schlessinger et al., 2010). The only common denominator between PEPT1 and the remaining transporters is the fact that their families (SLC15, SLC22 and SLC21/SLCO) all belong to the alpha-group of the major facilitator superfamily, which entails some level of homology in their type of folding (Höglund et al., 2011). It is known that PEPT1 binding relies on very specific three-dimensional requirements of distance and chirality(Foley et al., 2010) which may be difficult to properly portray even when using three-dimensional molecular descriptors. The use of predicted labels (prediction of binding to other transporters) as

additional features may supplement the molecular descriptors used in this work to aid a better prediction of PEPT1 uptake.

The SLC22 (where OCT1 belongs) and SLCO (where the OATPs belong) families show a relatively high degree of aminoacid sequence similarity (Schlessinger et al., 2013a, Schlessinger et al., 2010). These observed links between transporters in the CC model will be later explored in more detail.

Regarding the CC model’s accuracy, except for OATP1Aβ, which showed very low Spe, all

the other labels show acceptable predictive performance despite the high degree of class imbalance for most transporter labels. It should be noted that OATP1A2 has been associated with the largest chemical space among OATPs (Tirona and Kim, 2014), which potentially makes the task of modelling its substrate recognition patterns more difficult. The sheer quantity of pattern types is perhaps larger than the instances occurring under each pattern, which makes for a much shallower input provided to the learning algorithm. This possibly justifies the poor performance obtained for this transporter. Nonetheless, the obtained multi-label performance of the CC model (Table 5.4) showed a good level of quality with a 22.8% error rate (Hamming Loss) across all compound-transporter pairs, and an F1

score above 70%. However, as usual in any QSAR effort, all transporters’ models have an

overall higher propensity for correctly predicting substrates than non-substrates. This makes the CC model better suited for early screening (as opposed to late-stage prediction) where a higher false positive (false substrate) rate is less damaging to the long term success of the drug development project than the opposite scenario (higher false negative rate, which leads to false hits being moved forward across the pipeline).

Regarding the source of mispredictions in the CC model, one might consider the fact that the data used in the modelling comes from different sources that employ a variety of experimental designs and analytical methods to assign a compound as a substrate or non- substrate. Different laboratories can yield large variations in transporter function and expression. Furthermore, differences in experimental conditions potentiate the variability of obtained results (Artursson et al., 2013). This generates noise, which hinders the ability of the algorithm to properly learn any patterns between structure and response. Furthermore, as pointed out by Tu et al (Tu et al., 2013), any transporter substrates capable of high passive permeability will not be detected as substrates in permeability experiments. Consequently, actual substrates will be wrongly assigned to the non-substrate class, which will hinder the learning task as similar structure patterns will be found in both the non- substrate and substrate classes. In this work, where a compound had been reported as both as substrate and non-substrate of a transporter by different literature sources, the

compound was assigned to the substrate class following the principle of minimum evidence of substrate capability (i.e. even one single positive evidence of substrate capability shows that a compound is a substrate under certain experimental conditions). There was a total of 5 observations in the test set with such conflicting literature class assignments, among which only one was mispredicted under the CC model.

5.3.2. Model Validation

The nature of the automated exhaustive search through the different classifier chain orders does not accommodate an in-process trimming of potential outliers detected through AD analysis, before a set of predictions is fed forward across any given classifier chain. Furthermore, even if such in-process control was feasible, it would likely increase the degree of overfitting as outlier removal would be guided by an AD established using the validation set. This would be, to some extent, as controversial as simply removing mispredictions from the prediction set of a label before it is fed to the following labels. As a result of this, the AD analysis established for the best model (CC) is reserved to a prospective use, where it will aid the prediction acceptance decision. The contribution of outliers fed to any of the following labels during the training of a chain will likely be picked up during AD analysis of external predictions, as decision paths constructed from such outliers will be associated with a higher level of disparity between ensemble models. As a direct result, new instances that use such decision paths will be more likely to fall outside the AD for an established acceptance threshold (using the STD method). The established AD profiles (Appendix II, Figure A2.1-6) should be used with care as the amount of data in the test set of some transporters is so small that it might not convey representative accuracy values. This can be seen in the OATP2B1 model, where the level of precision does not correlate with accuracy. Examples such as this one are extremely challenging to characterize with respect to their AD. Nonetheless, every established AD allows the selection of a subset of the data associated with a reduced risk of mispredictions (even OATP2B1 where the lowest STD threshold shows the lowest error rate).

Lastly, in order to properly gauge the value of the built models, their prediction performances have to be evaluated in light of the content in activity cliffs. Knowing that these are located in areas where the structure-response relationship is more complex and changes unpredictably (Maggiora, 2006) or they result from incorrect class assignments (Sedykh et al., 2013), one would expect that the machine learning algorithm will mispredict them. Results (Appendix II, Table A2.1) show PEPT1 data contained a considerable amount of activity cliffs, which also indicates it is a challenging label to model as discussed before.

Other than PEPT1, all other transporters were associated with a relatively low number of activity cliffs, in comparison with the number of mispredictions. This shows that activity cliffs content is not likely to have considerably hindered the learning tasks.