Anexo I (Material y Métodos)

Allen et al first constructed structural alert-based models for Bowes’ targets in 2016,43 _and

updated the models in 2018.44 _{The updated models are similar to models developed in this work}

as both use an iterative cycle of creating structural alerts and removing chemicals from the training set. A major difference between the method presented here and Allen’s approach is a consideration of substructures’ occurrence in the inactive chemicals, not just in the active chemicals. The use of Bayesian statistics allows both variables to be considered, so false positives in the training set can be minimised whilst trying to build a model which correctly predicts true positives.

Allen has constructed models for the twenty-four biological targets for which models have been built in this work. However, the models have been constructed using different data sets so direct comparison is difficult.

Allen’s model is built on ChEMBL data only. This is unbalanced data, with most biological targets having significantly more actives than inactives. In construction of Allen’s models, the production of structural alerts from the training set does not consider occurrence of substructures in the inactive chemicals, so the imbalance in data is unimportant. However, an assumption is made to add inactive chemicals to the test set for the purpose of validation. For each target, chemicals which have been tested at other Bowes targets, but not the target of interest, are assumed to be inactive, giving approximately 11 000 additional negatives for each receptor. Such an assumption is likely to be valid for pharmaceutical chemicals because they are likely to have been tested at all targets during trials, but only active bioactivity data tends to be report in publications (from which ChEMBL collects data). However, the negative assumption will not be valid in all cases and there will be uncertainty about all assumed negatives. This uncertainty is not a major problem for Allen’s methods, where assumed negatives are used only for validation. The new approach, however, uses both training set positives and negatives in selecting the best substructures to be structural alerts. Uncertainty in negatives would therefore have a direct effect on model construction and could be problematic. Hence, importance of constructing new data sets with no assumptions regarding inactive data.

As Allen’s models and the new models are built and tested on different data sets, direct comparison is difficult. Both data sets take the majority of their active chemicals from ChEMBL, and the methods are both structural alert-based, so some comparisons should still be made. The average performances of the models in their respective test sets are shown in Table 2.6.

Table 2.6: Comparison of the performance metrics of Allen’s updated models from 201844 _{and the}

models developed in this work, labelled “Wedlake” models. “Screening” models are designed to have as large a sensitivity (SE) as possible, whilst the “Risk Assessment” models are designed to have the best overall performance. Performance metrics are calculated across the twenty-four Bowes targets for which both methods have created models. Different data sets are used for the Wedlake and Allen models and so direct comparisons of performance are difficult. However, Matthews Correlation Coefficient (MCC) is commonly considered the best single measure of overall model performance and a clear increase can be seen in both Wedlake models compared to Allen models. Both Wedlake models have higher SE than both Allen models. Allen models have higher specificity (SP) and accuracy (ACC), but these values are inflated by the use of assumed negative data. Overall, the performance statistics suggest the Wedlake models are significant improvements on the Allen model.

Allen’s models are tested using the assumed negatives, hence the large excess of negative data. For these unbalanced data sets, MCC provides the fairest indication of overall performance. The MCCs of both new models are significantly higher than the Allen model, indicating better overall performance of models.

The specificity of both Allen models is higher, but there is uncertainty with the negative predictions due to the inclusion of assumed negatives. The vast number of the assumed negatives dominates the accuracy, resulting in higher, potentially misleading accuracy values for the Allen models. Sensitivity of the new models is greater than that of the Allen models.

In the Allen models the Screening model differs from the Risk Assessment model by using less stringent minimum requirements for structural alerts, resulting in a greater number of alerts being used to achieve a larger sensitivity. In contrast, in the new models the Screening model uses fewer structural alerts than the Risk Assessment model to achieve a greater sensitivity. The same minimum requirement for structural alerts is used in both models. Instead, other parameters are changed, resulting in different substructures being chosen as structural alerts. The Screening model contains larger, less specific structural alerts which are contained by a greater number of actives.

Model Alerts TP FP FN TN SE SP ACC MCC

Allen Risk Assessment 53.0 250 153 109 15465 64.4% 98.7% 97.9% 0.646

Allen Screening 86.7 308 572 51 10698 80.9% 94.8% 94.6% 0.585

Wedlake Risk Assessment 86.5 532 38 78 460 86.6% 90.9% 90.2% 0.782

Wedlake Screening 42.5 559 77 51 421 90.6% 82.4% 88.9% 0.748

Overall, the new models represent a significant improvement on the Allen models in terms of performance. The use of Bayesian statistics to pick structural alerts allows the consideration of the occurrence of a substructure in both the active chemicals and in the inactive chemicals. The new, balanced data sets constructed in this work mean that no assumptions about data need to be made.