There were 12 epidemiological model articles identified, including 9 papers that published original models. The identified epidemiology models are the Bach, Liverpool Lung Project, Spitz, African-American, PLCO,
PLCOM2012, PLCOM2014, Hoggart and Pittsburgh Predictor models.
These models are presented separately including their validation results. The model summaries and validation results are also present in 4 tables; these present the model variables, building dataset description, model results and a summary of model validation tests conducted.
3.5.1 Liverpool Lung Project Model
The Liverpool Lung Project (LLP) (Cassidy et al 2008) model was published in 2007 to estimate the risk of lung cancer incidence occurring in an individual within the next five years. Risk of developing lung cancer was modelled by a multivariate conditional logistic regression. The model was devised in a UK case-control population matched by age and gender. The model uses UK population specific lung cancer incidence rates stratified by age and gender in the prediction model. This could limit the model’s success in different countries where the incidence rates may differ. However, considering the population specific age and gender incidence rates aims to limit concerns with matching over these variables. The model also considered diagnosis of a prior malignant tumour, smoking duration, pneumonia, asbestos exposure and family history of lung cancer. The model is only applicable to people aged 40-80 years as the incidence rates stratified by age and gender were only provided for this age range. The model is applicable to all people in this age range; whether they are never- or ever-smokers [72].
The model was internally validated in the article using a 10-fold cross validation in an attempt to eliminate optimism [72]. The internal validation reported an AUC of 0.71 which was a reasonable dis- crimination. The internal validation also evaluated the prediction rules. These were reviewed at 2.5% and 6% risk thresholds and the sensitivity and specificity were reported. At 2.5% threshold the sensitivity and specificity results were 0.62 and 0.70 respectively. These results are promising, which was expected as the model reported a good overall discriminative ability, highlighted by the AUC result. At the 6% threshold the sensitivity was 34% while the specificity was 90%. This would allow the prediction model, if implemented as a selective screening tool, to remove a high proportion of lung cancer free individuals from unnecessary screening; a benefit if the screening programme objective was to reduce unnecessary screening costs. The ability to still capture 34% of individuals with lung cancer while limiting screening of disease free individuals is an advantage of the model. However, the results should be verified in an external validation to allow a stronger understanding. Additionally, validating the model in a case-control population often leads to enhanced validation results as the cases are commonly at an advanced stage and have an enhanced risk.
The LLP model was validated in five distinct datasets and the results showed a similar level of per- formance to the internal validation. One external validation [69] evaluated the model performance using 4,900 case-control study participants in a US sample population. This provided an opportunity to eval- uate how the model performed in a distinct population while considering UK age and gender specific lung cancer incidence rates. The AUC had a slightly poorer performance at 0.69 in comparison to the internal validation; however, the model still performed well when considering the prediction rules. The sensitivity and specificity rates were reported at the 2.5%, 5% and 7.5% thresholds for 5-year risk. The model reported similar results at the 2.5% risk threshold as the internal validation, with a sensitivity and specificity both of 0.67. This represents a good trade-off between the sensitivity and specificity highlighted by a Youden’s (J index) score of 0.33 in comparison to 0.32 in the internal validation. At 5% risk, the sensitivity dramatically reduced to 45.5% meaning over half of all individuals with lung cancer would not be screened and the J index reduced to 0.304. This trend continued at 7.5%, where a very low sensitivity of 31.2% was reported, although the specificity exceeded 90%, which resulted in a J Index of 0.235. The external validation demonstrates that at the low 2.5% threshold the model would be able to capture a high proportion of individuals who develop lung cancer, which may improve diagnosis and survival rates for lung
cancer. In contrast, if a selective screening programme had a limited budget, then a high risk threshold of 6% (internal validation) or 7.5% would allow the model to avoid screening over 90% of controls.
The LLP model was validated in another article [73]; where the model was validated in three large datasets. The studies collected were case-control and prospective population cohorts that were based across Europe and North America. The results varied between studies but suggested that the LLP model
has strong potential as a selective screening criteria. The three AUC results were 0.67, 0.76 and an
impressive 0.82 in the large population-based prospective study cohort of 7,652 patients. However, this was a Liverpool population similar to which the model was devised. In this cohort the sensitivity and specificity rates at the 2.5% threshold were 74.3% and 67.4% respectively, which combine to give the highest recorded J-Index for the LLP Model at 0.42; a very strong result. The model also reported a model accuracy of 67.8% in the cohort. The results were very promising and the model’s ability to perform to a high standard in a population cohort demonstrates its potential for a selective screening trial. The model reported good results in the other studies although they were not as high, which are presented in Table 3.3. The difference between the three datasets indicate the model cannot perform to the same standard in populations outside of Liverpool or the UK.
One final external validation was conducted [74]. This used a case-control study with 1,275 participants and reported the sensitivity and specificity rates. The results were slightly poorer and were validated at the 5.12% risk threshold. Here, the model reported a sensitivity of 49.9% but a specificity of 79.8%. The calibration was not formally reported but the model recorded a “moderate overall calibration and improved accuracy at higher values of predicted risks” [74].
Based on the success of the LLP Model in validation studies the model has been evaluated as a selective screening tool in a screening trial. The trial is reported in Section 1.8.2. Participants were considered to be at high risk and recommended for screening, if their risk exceeded 5%. At this risk threshold the model would maintain a specificity of approximately 45-50% for a specificity around 75-80% based on reported validated results. This level of performance would result in the LLP Model reporting a Youden’s Index 0.25, which is a good result.
Across studies, the AUC was reasonable with internal and external results ranging from 0.67 to 0.76 indicating a good discrimination. This was surpassed by an AUC of 0.82 in one validation, although the validation and model building data were both samples from Liverpool populations where the UK age and gender specific incidence rates are relevant. A slightly lower performance was observed in validations conducted outside of the UK. Based upon these findings the model should be tested in non UK cohorts to assess how successfully these can be used in different populations. Additionally, the calibration has not been adequately tested and needs to be more comprehensively assessed. At this stage it is clear that the model has potential but there is room for improvement and more thorough validations to assess the calibration and differences between UK and non-UK populations.
3.5.2 Spitz Model
The Spitz Model (Spitz et al 2007) predicts one-year absolute risk of lung cancer. The logistic regression model calculates an individual’s likelihood of developing lung cancer within one year and the competing risk of mortality without a lung cancer diagnosis. The model is applicable to anyone; never- and ever-smokers. While individuals are required to be at least 20 years of age, lung cancer rarely presents at younger ages so this is not a limitation of the model. Therefore, versatility of the model to be applicable to everyone older than 20 years of age is a key advantage.
The model was devised with 3,852 case-control participants matched by age, sex and smoking status collected from the USA [75]. To avoid concerns with matching; age and gender lung cancer incidence and death rates were incorporated into the model by using the SEER rates. The SEER rates are the observed incidence rates across the USA. Smoking status is incorporated into the model using gender-smoking incidence groups.
The dataset was split into a model building set (75%) and an external validation set (25%) and three distinct models were devised for never, former and current smokers. These consider different variables in
different logistic regression models. Ideally, the model building dataset would include all the participants to allow as much evidence as possible to be included when devising the model. Then the Spitz Model could be validated internally and subsequently assessed in external validations.
The calibration and discrimination were measured in the external validation in the original article [75]. The validation results were presented separately for the never-, former- and current-smoking models. For the never-smokers model the model reported a good calibration with a Hosmer-Lemeshow p-value of 0.777. However, the never-smoker model reported a poor AUC result of 0.57 (95% CI [0.47, 0.66]) which includes 0.5; suggesting the model has no better than chance in assigning a higher risk to an individual with lung cancer than an individual who is disease free. The results slightly improved in the former-smokers model with a good calibration (p-value 0.712) and an improved AUC of 0.63 (95% CI [0.58, 0.69]). However, the AUC result does not suggest the model is a robust tool to distinguish between individuals with or without lung cancer. Finally, the current-smokers model was also well calibrated (p-value 0.688) but reported another poor AUC of 0.58 (95% CI [0.52, 0.64]). Across the three distinct models for different smoking statuses there was a reoccurring pattern. The models were well calibrated indicating that the model could be used to provide accurate risks for individuals. However, the model reported a very poor discriminative ability suggesting the model would have a limited ability to identify high risk participants for screening. However, this could be more extensively validated by evaluating the prediction rules, which was not conducted in the article in which the Spitz Model was presented.
The Spitz Model was externally validated [69] but the model was extended to predict absolute risk over 5-years by “combining the risk of cancer from the relative risk model with age and gender incidence and mortality models recursively five times” [69]. In the 4,900 case-control population the model reported a much improved discrimination with an AUC of 0.69 (95% CI [0.66, 0.71]). This would suggest extending the Spitz Model duration may improve the model’s discriminative ability as it can more robustly distinguish between individuals with and without lung cancer. This could be due to the original 1-year time period being too small as the majority of individuals, both diseased and disease free, have low risks of developing cancer over such a small time period.
Despite the improved discriminative ability, the prediction rules results were surprising. The results at the 2.5% risk threshold suggest this was too high with the majority of the participants being eliminated as shown by a low sensitivity of 26.6% and a specificity rate of 94.4%. Decision makers would need to decide whether the low sensitivity is beneficial as a trade-off of eliminating a high proportion of unnecessary screening with the high specificity. To allow an informed decision the prediction rules should be evaluated at a lower risk threshold. The PPV at the 2.5% risk threshold was 0.882 and the NPV was 0.45, this is a very good PPV, although a case-control population favourably reflects these results due to the high incidence rates in the validation set that would not be observed if implementing the model as a selective screening tool.
The final external validation of the Spitz model used 1,340 patients in another case-control study [76]. This study reverted back to applying the Spitz Model to predict the absolute one-year risk. The discrimination was measured and the model reported AUC values of 0.67 and 0.68 for former- and current- smokers respectively, which indicates a reasonable discriminative ability. Unfortunately, the prediction rules were not assessed in the paper.
Current reporting for the Spitz Model has been subpar with only the AUC results being consistently reported. Indeed, the optimal duration for the prediction model has not been determined at this stage with validations alternating between applying the model to predict 1 or 5-year absolute risk. The AUC results for the Spitz Model varied from very poor to reasonable but further testing needs to be conducted. This includes reviewing the prediction rules, to offer a more comprehensive review of the model’s potential, which currently have only been evaluated in one article. The wide applicability is a clear strength of the model, which would suggest if a robust optimal risk threshold can be identified then the Spitz Model may be considered to identify a target population for screening. Additionally, the model was well calibrated, when validated and based upon these results the model could be made available to the public to calculate their risk and increase awareness.
Variables Bac
h
LLP Spitz Afr.-Amer. PLCO PLCO
12/14 Hoggart Pittsburgh Personal Information Age X X X X X X X X Gender X X X X Ethnicity X
Body Mass Index X X
Education X X
Prior Malignant Tumour X X
X-Rays X Smoking History Smoking Status X X X X X X Start Age X Cessation Age X X Smoking Duration X X X X X X CPD X X X X Pack Years X X X Quit Duration X X X X ETS X
Family History of Cancer
Cases of Smoking Related Cancer X
Cases of Lung Cancer X X X X
Age of Onset of Lung Cancer X
Exposures and Conditions
Asbestos Exposure X X X Dust X X Hay fever X X Emphysema X COPD X X X Pneumonia X X
The following models have restrictions and designs;
Applicable to Never Smokers X X X X M2014
Applicable to Ever Smokers X X X X X X X X
Age 50 - 75 40 - 80 20+ 20+ 35+
Smoking History 30 PY
Predicts Incidence X X X X X X
Predicts Survival X X X X
Risk Length (Years) 1+ 5 1+ 5 9 6 1+ 6
3.5.3 Bach Model
The Bach Model (Bach et al 2003) was devised using a large 18,172 cohort collected in the USA [77]. The model estimates an individual’s absolute risk by combining two separate one-year logistic regression models that calculate “risk of developing lung cancer” and the competing “risk of dying without a lung cancer diagnosis”. The model duration can be extended by running the models recursively and the model was designed to predict absolute risk for 10 years [77].
The Bach Model is quite restrictive and can only be applied to a target population aged 45–69 years who are ever- smokers with a minimum 30 pack year smoking history and former-smokers are required to have quit within the last 15 years [77]. The restrictive population lowers the model utility as the 15% of lung cancer incidences that occur in never-smokers [4, 5] and further incidences that occur in lower smoking ever-smokers will not be identified. This lowers the impact of the model as an early detection tool as a sizeable proportion of lung cancers would not be identified in a selective screening trial.
The Bach Model was validated in a 300 participant cohort from the Mayo Clinic, USA in the original article. The Mayo Clinic entries satisfied the original criteria as “subjects must be aged 55–74 years, have smoked a minimum of 30 pack-years and be current smokers or former smokers who quit within the last 15 years” [77]. The validation compared the estimated one-year absolute risk and observed incidences over the duration. The study found the participants with the lowest 25% of risk contained only 8% of incidences whilst the top risk quartile contained 50%. While this demonstrates the model’s ability to assign higher risk to individuals who develop lung cancer the risk thresholds at these quartiles were not reported which may indicate how the model could be applied as a selective screening tool. The AUC result (0.72) supported the suggestion the model had a reasonable discriminative ability. Unfortunately, no tests were conducted to evaluate the prediction rules and the model’s clinical utility.
The Bach Model was externally validated in “Validation of a Model of Lung Cancer Risk Prediction among Smokers” [78] with 6,239 smokers recruited by the Alpha-Tocopherol, Beta-Carotene Cancer Pre- vention (ATBC) Study. The study recruited some participants outside the range specified in the original article; men aged “50-69 who had smoked 5 or more cigarettes a day” [78]. This tested the model’s adaptability in different environments, in this case for lower ever-smokers. The expected incidences in the cohort over 10 years was 297.07 in comparison to the observed 333. This equates to expecting 89% of the observed incidences. The under estimation could indicate the model struggled to estimate risk in the lower risk participants with a reduced smoking history. The validation study only estimates total group deaths rather than formally assessing the calibration, discrimination and prediction rules. The study could be improved by splitting the validation to assess performance for patients inside and outside the original entry criteria to determine if the model’s target population could be expanded.
Finally, the model was externally validated in an article that compared the Bach, LLP and Spitz models [69]. The Bach Model predicted absolute risk over 5 years in the study. The study included a large range of participants but non-smokers were excluded since their risk is not calculable by the model. The AUC was 0.66, indicating a reasonable discriminative ability and the study reported no significant difference in the AUC when stratified between age groups. However, this was the lowest AUC of the models considered in the same dataset (Spitz and LLP). This was reflected in the sensitivity and specificity rates with a highest J-Index of only 0.19 at the 2.5% risk. The poor results could indicate that the model has difficulty in predicting risk in less intense smoking populations and when applied for a shorter time frame than the 10 years original proposed.
In summary, the Bach Model is limited by only considering ever-smokers. The current validation testing has been poor; the discrimination, calibration and prediction rules have been infrequently recorded so it is difficult to judge the model’s performance. The preliminary testing indicates that the Bach Model may not be as robust a selective screening tool as the LLP and Spitz models. Further testing is required to