COSTOS NO ASEGURADOS O VARIABLES

are presented in Table 11.1.

Model Weightings

With Bach Without Bach

Bach 0.25271 NA

PLCO 0.25201 0.33766

Hoggart 0.24460 0.32651

Pittsburgh 0.25068 0.33582

Table 11.1: Model Weightings from Model Averaging

For both versions the final weights given to the separate models was very similar. This may be a result of all the models being poorly calibrated even after the recalibration. This is most likely a consequence of the high lung cancer incidence rate in the model aggregating dataset. Therefore, the method could not identify the better calibrated models to assign a higher weight.

The new models were validated to assess if model averaging created a more robust prediction model.

Bach Validation Internal/External Hos.-Leme. Brier Score AUC [95% CI] Threshold (%) Sens. Spec. Youden PLR With 1 Internal 0.3617 (277.63) 0.4863 0.5337 [0.512, 0.556] 1.05 28.444 82.980 0.1142 1.6713 External 0.4776 (416.96) 0.2859 0.6239 [0.605, 0.643] 0.76 99.967 0.000 -0.0003 0.9997 2 Internal 0.3617 (277.63) 0.4863 0.5337 [0.512, 0.556] 1.06 19.556 89.993 0.0955 1.9541 External 0.4776 (416.96) 0.2859 0.6239 [0.605, 0.643] 0.67 19.056 90.026 0.0908 1.9105 Without 1 Internal 0.0001 (648.84) 0.4753 0.5920 [0.577, 0.608] 1.41 84.755 31.648 0.1640 1.2400 External 0.0003 (855.97) 0.3873 0.6808 [0.668, 0.694] 1.23 100.000 0.037 0.0004 1.0004 2 Internal 0 (669.64) 0.4705 0.5801 [0.563, 0.597] 1.33 12.294 89.987 0.0228 1.2278 External 0.0527 (779.39) 0.3885 0.6883 [0.674, 0.702] 1.08 25.399 89.982 0.1538 2.5353

Validation 1; Optimal Risk Threshold - Validation 2; UKLS Guidelines

The calibration results for the meta-model including the Bach Model was promising. The meta-model was well calibrated in both the internal and external datasets (Table 11.2), most likely a consequence of recalibrating the models before aggregating them. However, the promising calibration results are not replicated in the meta-model excluding the Bach Model, while the calibration improves, highlighted by the Hosmer-Lemeshow chi-squared and the Brier Score results, the model did not report a good calibration.

Unfortunately, the promising calibration improvement was not reflected in an improved discriminative ability or prediction rules performance. Indeed, the proposed aggregated model performs poorer than the original models.

In summary, Model Averaging is a simple method to apply and distinct models can be combined. However, when the original models are poorly calibrated, even after recalibration, then the weights assigned to each model in the final meta-model are very similar. While this method notably improved the model calibration, to the extent the meta-model with the Bach Model was well calibrated in both the internal and external validation, the discriminative ability of the new model was poorer than the original models. While Model Averaging can create a well calibrated model, if the model updating datasets has a high disease prevalence rate, as observed in this instance, then the proposed meta-model would be inappropriate in populations with more realistic incidence rates. Overall, Model Averaging failed to improve the prediction rules in comparison to the original models and a more robust lung cancer prediction model was not created.

11.4

Method Two - Bayesian Model Averaging

The next approach is Bayesian Model Averaging (BMA) with a non-informative prior. There are similarities between this method and Model Averaging, the models are applied separately and then weighted in the

meta-model. The models are assigned a weight based on their calibration in the dataset. However,

the weights are calculated slightly differently and the models are not initially recalibrated unlike Model Averaging. The original models were poorly calibrated in the dataset due to the high incidence rate. Not recalibrating the model originally could see some of the models being penalised in the final meta-model while actually being successful models.

For each model the likelihood function in the dataset was calculated in the same way as Model Averaging (Equation 11.5), as follows; Li = N X j=1 (yj× ln (mj)) + ((1 − yj) × ln (1 − mj)) (11.5) where yj = ( 0, if participant is a control 1, if participant is a case (11.6)

Where, mj is the predicted risk for participant j in the model.

After calculating the likelihood function, Li, for each model the following calculation was performed,

as presented in Equation 11.7. To perform this calculation the model dimension, dj, is required, which is

the number of distinct variables in the model;

Mi= e



Li+di₂ln(N )



(11.7) Here, N is the number of participants in the dataset which is fixed because they are applied in the same sample population for which all the models are applicable. For the lung cancer prediction models their model dimensions are as follows;

Model Model Dimension

PLCOM2014 12

Hoggart 12

Pittsburgh 4

Bach 7

Table 11.3: Lung Cancer Model Dimensions

Now, Mi can be determined and the final model weightings calculated;

wi =

Mi

Pn

j=1Mj

(11.8) This was a straightforward method to combine multiple lung cancer prediction models. The weighting applied to each model to produce a final lung cancer risk are as follows, for both options including or excluding the Bach Model;

Model Weightings

Model with Bach Model Model without Bach Model

Bach 2.585E-09 NA

PLCO 1.852E-03 0.500002

Hoggart 0.99815 0.499998

Pittsburgh 7.292E-17 1.86E-18

Table 11.4: Model Weightings from BMA

The two different versions led to unusual model weights. When considering the Bach Model, the model weightings does not best synthesis the evidence from the distinct models by minimising the impact of some models in the meta-model. Indeed, the final model was effectively the Hoggart Model which was assigned a weight exceeding 0.998. The remaining models were assigned weights very close to zero. This demonstrates how BMA can assign excessive weights to some models because the approach assumes some models are correct and the remaining models should be penalised. When excluding the Bach Model from BMA the model weights were more evenly distributed. The meta-model is effectively a combination of the

PLCOM2014 and Hoggart models, both assigned a weight of effectively 0.5. The Pittsburgh Model was still

minimised in the final model.

Bach Validation Internal/External Hos.-Leme. Brier Score AUC [95% CI] Threshold (%) Sens. (%) Spec. (%) Youden PLR With 1 Internal 0 (494.26) 0.3726 0.5295 [0.508, 0.551] 25.05 29.63 76.48 0.0611 1.2597 External 0.0021 (503.49) 0.5148 0.6103 [0.591, 0.629] 24.98 31.14 81.01 0.1215 1.6396 2 Internal 0 (494.26) 0.3726 0.5295 [0.508, 0.551] 32.24 12.59 89.99 0.0259 1.2583 External 0.0021 (503.49) 0.5148 0.6103 [0.591, 0.629] 32.50 18.66 90.03 0.0869 1.8708 Without 1 Internal 0 (726.68) 0.4220 0.6114 [0.596, 0.627] 1.50 82.86 36.21 0.1907 1.2990 External 0 (998.86) 0.5307 0.6922 [0.680, 0.705] 1.50 84.06 42.15 0.2622 1.4532 2 Internal 0 (735.91) 0.4152 0.5981 [0.581, 0.615] 15.97 12.67 89.99 0.0266 1.2654 External 0 (876.01) 0.5227 0.7000 [0.686, 0.714] 14.05 25.91 89.98 0.1589 2.5865

Validation 1; Optimal Risk Threshold - Validation 2; UKLS Guidelines

Table 11.5: BMA Validation Results

The meta-model including the Bach Model had a similar performance to the Hoggart Model, as ex- pected, when validated across all tests 11.5. When excluding the Bach Model in the meta-model, there was

an improvement in performance. However, this did not exceed the performance of the original PLCOM2014

over 6-years. Therefore, the meta-model to maintain a specificity of 90% would need to be applied at approximately the 15% risk threshold (Table 11.5). This may be inappropriate and suggests recalibration before applying the method may allow more appropriate risks to be developed. However, this may not be advisable in a dataset with an unrealistic lung cancer incidence rate because the recalibrated models may estiamte even further distorted risks.

In summary, there was a contrasting performance between the two meta-models created by BMA. When considering the Bach Model the method’s limitations are highlighted as all except one model are negated in the final meta-model. When excluding the Bach Model the weighting is slightly less biased but the Pittsburgh Model is still minimised in the final model. The method may benefit from initially recalibrating the models to allow a fairer inclusion of the models. While the meta-model excluding the Bach Model had a reasonable performance this did not sufficiently improve upon the original models in the external validation so a more robust model was not created.