RIESGOS PSICOSICIALES

The original models are validated to provide a baseline performance for the updated and meta-models to be compared. The model updating datasets and the NY Wynder dataset are both external datasets for the original models. In contrast the newly updated models will be internally validated in the model updating dataset. Therefore, it is expected that the updated models will have an improved performance when internally validated. Therefore, the NY Wynder dataset will offer a more rigorous assessment of the updated models to assess if a more robust model has been devised.

9.5.1 Single Model Updating

There were concerns the high lung cancer incidence rates would hinder the PLCOM2014Model calibration.

Indeed, the Hosmer-Lemeshow test in the IPD returned a p-value below 0.001 and a χ2 exceeding 1,250.

The PLCOM2014 Model also reported a poor calibration in the NY Wynder dataset. The poor calibration

demonstrated through the Hosmer-Lemeshow test is supported by the Brier Score which exceeded 0.4. The original model reported a reasonable discrimination in the model updating dataset, with an AUC of 0.68, although this decreased to 0.66 when restricted to a sample population of participants aged 50-75 years. The model was optimal at the 0.316% risk threshold; here, the model reported a sensitivity of 78.33% and a specificity of 50%. The high sensitivity and specificity resulted in a Youden index just below 0.3 and a PLR of 1.58. When applying the model to the UKLS target population the prediction rule

performance, similar to the AUC, was lowered. The PLCOM2014Model should be applied at the 3.16% risk

threshold to maintain a specificity of 90%. In the IPD the sensitivity dropped below 20%, a poor capture rate highlighted by a Youden index below 0.1 and a PLR of only 1.96 despite the high specificity.

Validation Internal/External Hos-Leme. Brier Score AUC [95% CI] Threshold (%) Sens. Spec. Youden PLR 1 Internal 0 (1266.26) 0.4007 0.6813 [0.669, 0.694] 0.32 78.33 50.32 0.2865 1.5768

External 0 (3428.03) 0.4887 0.7667 [0.757, 0.776] 0.32 86.85 54.68 0.4153 1.9163 2 Internal 0 (726.84) 0.4173 0.6567 [0.642, 0.671] 3.19 19.59 90.00 0.0960 1.9601 External 0 (9793.05) 0.4906 0.7834 [0.773, 0.794] 2.95 35.02 89.99 0.2502 3.5002

Validation 1 at the optimal risk threshold Validation 2 using the UKLS guidelines

Table 9.4: External Validation of PLCOM2014 Model in the Single Updating Datasets

The PLCOM2014Model performed better in the NY Wynder dataset. The AUC was 0.767 and reported

an encouraging prediction rules performance. At the optimal risk threshold, identified in the model up- dating dataset as 0.316%, the model reported an increased sensitivity of 86.85% and specificity of 54.68%. Therefore the Youden index was above 0.4. In the target UKLS population the performance improved fur- ther. The AUC increased to 0.783 and at a risk threshold of 2.95% the sensitivity was 35% for a specificity of 90%.

Overall, the model was poorly calibrated as was expected. It is likely that the recalibrated models will report a higher performance because they will predict the higher incidence rate. Additionally, the

PLCOM2014 Model had a limited discriminative ability in the model updating dataset. However, the AUC

and prediction rules performance improved in the NY Wynder dataset such that the new models will be required to produce an exceptional performance to demonstrate an improved model.

9.5.2 Model Aggregation without the Bach Model

The original models reported a poor performance in the model aggregation dataset. The high incidence rates resulted in all the models reporting a poor calibration measured by the Hosmer-Lemeshow tests and supported by a Brier score which was higher than 0.4 for all the models.

The models also reported a low discriminative ability in the aggregation dataset with all the models reporting an AUC of approximately 0.6. This was reflected in poor prediction rules results where none of the models achieved a Youden index exceeding 0.2 and the PLR was low with most results below 1.5. This poor performance was also observed in the UKLS target population.

Model Validation Internal/External Hosmer-Lemeshow Brier Score AUC [95% CI] Threshold (%) Sens. Spec. Youden PLR

PLCO2014 1 Internal 0 (687.51) 0.4666 0.6222 [0.607, 0.637] 0.82 72.00 45.34 0.1734 1.3172 External 0 (1059.44) 0.5917 0.6806 [0.668, 0.693] 0.82 75.58 49.57 0.2515 1.4988 2 Internal 0 (563.05) 0.4623 0.6148 [0.598, 0.631] 3.62 17.85 89.99 0.0784 1.7829 External 0 (1292.11) 0.5891 0.7016 [0.688, 0.716] 3.94 24.99 89.98 0.1498 2.4948 Hoggart 1 Internal 0 (948.39) 0.3874 0.6040 [0.589, 0.619] 2.24 81.20 37.19 0.1839 1.2929 External 0 (748.86) 0.4801 0.6778 [0.665, 0.691] 2.24 81.42 42.86 0.2428 1.4250 2 Internal 0 (795.28) 0.3792 0.5907 [0.574, 0.607] 29.05 12.58 90.03 0.0261 1.2615 External 0 (589.00) 0.4678 0.6819 [0.668, 0.696] 24.84 25.80 89.94 0.1574 2.5642 Pittsburgh 1 Internal 0 (744.99) 0.4638 0.5911 [0.576, 0.607] 0.72 83.14 33.69 0.1682 1.2537 External 0 (370.90) 0.5887 0.6793 [0.667, 0.692] 0.72 81.51 43.91 0.2542 1.4531 2 Internal 0 (672.30) 0.4579 0.5751 [0.558, 0.592] 5.12 14.93 89.33 0.0426 1.3988 External 0 (292.67) 0.5842 0.6951 [0.681, 0.709] 4.66 24.13 89.98 0.1411 2.4085

Validation 1 at the optimal risk threshold Validation 2 using the UKLS guidelines

Table 9.5: External Validation of Models in the Datasets without the Bach Model

The performance improved in the NY Wynder dataset although the models were still poorly calibrated. The AUC increased to between 0.67-0.70 for the models and this improvement was observed in the predic- tion rules performance. At their optimal risk threshold, the models reported a Youden index of 0.25. The models reported a sensitivity of approximately 25% when the specificity was fixed at 90% in the UKLS target population.

In summary, the models were poorly calibrated and reported a similar discriminative ability. This is likely to reduce the difference in weightings between the BMA methods because the additional weighting

will not allow a drastic change from the non-informative prior. Additionally, basing the aggregation on the calibration may be unreliable since the models were poorly calibrated, particularly for methods that do not originally recalibrate the model. In the NY Wynder datasets the models will aim to eclipse an AUC of 0.70, a Youden index of 0.25 at the optimal risk threshold, and a sensitivity of 25% using the UKLS criteria.

9.5.3 Model Aggregation with the Bach Model

In the final validation the models were poorly calibrated in both datasets as a result of the high lung cancer incidence rate. The poor calibration is supported by the Brier score, which is above 0.4 for all the models.

The models also had a poor discriminative ability in the aggregation dataset. The PLCOM2014 Model

marginally reported the highest AUC with a result of 0.59, highlighting the low discriminative ability standard of the models in the dataset. Further the Bach Model reported an AUC below 0.5.

Model Validation Internal/External Hosmer-Lemeshow Brier Score AUC [95% CI] Threshold (%) Sens. Spec. Youden PLR

PLCO2014 1 Internal 0.1283 (296.53) 0.4682 0.589 [0.568, 0.611] 2.26 51.85 62.24 0.1409 1.3731 External 0.0890 (455.37) 0.6757 0.597 [0.578, 0.616] 2.26 57.69 55.59 0.1329 1.2993 2 Internal 0.1283 (296.53) 0.4682 0.589 [0.568, 0.611] 4.42 18.96 89.99 0.0896 1.8949 External 0.0890 (455.37) 0.6757 0.597 [0.578, 0.616] 5.40 18.43 90.03 0.0845 1.8476 Hoggart 1 Internal 0 (460.43) 0.3725 0.529 [0.508, 0.551] 24.96 29.70 76.33 0.0604 1.2551 External 0.0001 (251.50) 0.5146 0.609 [0.590, 0.628] 25.01 31.14 81.01 0.1215 1.6396 2 Internal 0 (460.43) 0.3725 0.529 [0.508, 0.551] 32.29 12.37 90.07 0.0244 1.2452 External 0.0001 (251.50) 0.5146 0.609 [0.590, 0.628] 31.25 18.76 90.03 0.0878 1.8807 Pittsburgh 1 Internal 0 (550.24) 0.4631 0.526 [0.504, 0.547] 4.66 26.37 82.18 0.0855 1.4796 External 0.0001 (251.50) 0.6696 0.611 [0.592, 0.630] 4.66 20.94 88.81 0.0975 1.8714 2 Internal 0 (550.24) 0.4631 0.526 [0.504, 0.547] 5.63 10.81 94.30 0.0512 1.8981 External 0.0001 (251.50) 0.6696 0.611 [0.592, 0.630] 5.63 15.62 91.85 0.0747 1.9161 Bach 1 Internal 0.1529 (293.8) 0.4577 0.554 [0.533, 0.576] 2.12 76.30 32.36 0.0866 1.1280 External 0.0001 (251.50) 0.6614 0.601 [0.582, 0.620] 2.12 76.39 36.51 0.1290 1.2032 2 Internal 0.1529 (293.8) 0.4577 0.554 [0.533, 0.576] 6.87 13.56 89.99 0.0355 1.3546 External 0.0001 (251.50) 0.6614 0.601 [0.582, 0.620] 7.01 18.53 90.03 0.0855 1.8575

Validation 1 at the optimal risk threshold Validation 2 using the UKLS guidelines

Table 9.6: External Validation of Models in the Datasets with the Bach Model

The models improved in the NY Wynder dataset with no leading model being identified because they all reported an AUC of approximately 0.66. The prediction rules were similar for the models also with a Youden index of 0.12 at the optimal risk threshold. Evaluating the models using the UKLS guidelines a sensitivity around 18% was reported for all models. This is the baseline performance upon which the meta-model will attempt to improve upon.

There are concerns the poor calibration may hinder the aggregation methods based on this evidence, and the poor AUC across all the models may not provide sufficient information for the additional AUC weighting for BMA.

9.6

Summary

The chapter presented the methodology to review both the original models and the new models. A dataset analysis of the model updating and the NY Wynder datasets was conducted. The original models were then validated. The calibration was poor in the datasets due to the high lung cancer incidence rates observed. This was identified as a concern for single models updating, as the new models will be recalibrated to reflect the high lung cancer incidence rate. The proposed models then may be unrealistic when applied in a population with a more realistic lung cancer incidence rate. Additionally, the calibration information may be unreliable on which to base the model aggregation.

The PLCOM2014 Model will be updated using single model updating techniques. In the external vali-

sensitivity of 35% using the UKLS guidelines, for which the updated models will attempt to surpass. For model aggregation excluding the Bach Model the baseline performance in the external validation which the meta-models should aim to surpass was an AUC of 0.70, a Youden index of 0.25 at the optimal risk threshold, and a sensitivity of 25% for the UKLS guidelines. Finally, when considering the Bach Model, the proposed meta-models will attempt to eclipse an AUC of 0.66, a Youden index of 0.12 at the optimal risk threshold, and a sensitivity of 18% in the UKLS target population in the NY Wynder dataset.

The meta-model including the Bach Model may allow a better evaluation of BMA with an additional weighting based on the AUC as there was more disparity in the AUC results across the models.

The next stage of the project will apply the methods for the single model updating and the model aggregation to devise new models. These will then be validated and compared with these original model results, to assess if an improved lung cancer prediction model has been devised.

CHAPTER

10

Updating a Single Lung Cancer Prediction Model

10.1

Introduction

The literature review identified practical methods to update a single prediction model. These methods

will be applied to the PLCOM2014 Model, which was selected because the model demonstrated the leading

performance in the external validations. The new updated models will be presented and externally validated to assess if the methods have created an improved model that could be considered as a clinical utility. This will also identify successful methods which could be used for prediction models of different diseases or prediction models that consider clinical markers once these models demonstrate an improvement over epidemiological prediction models.

10.2

Objectives

The chapter will apply updating methods to the PLCOM2014 Model in an attempt to create an improved

lung cancer prediction model. The chapter objectives are;

1. Apply the identified methods from the literature review to update a single prediction model to the

PLCOM2014 Model.

2. Present the updated model coefficients.

3. Externally validate the model in comparison to the original model to assess if an improved lung cancer prediction model was devised.

4. Present the Stata code and screening guidelines for any model shown to have an improved perfor- mance.

5. Review when methods are appropriate to update a prediction model.