This review focused on published prediction model study articles which describe the development of prediction models in clinical settings where competing events are
a likely issue. The prediction model development studies were identified from a subset
of systematic review articles identified in Chapter 2. Initially, the systematic reviews examined in Chapter 2 were screened to ascertain which were likely to contain prediction models affected by competing events. Consequently, only those 19 systematic reviews categorised with a high potential for competing risk bias were considered. An additional list of diseases and population characteristics, suggested to identify populations which are susceptible to competing events (Koller et al., 2012), presented in Table 3.1, was applied to further reduce the subset of systematic reviews.
Table 3.1: List of diseases and populations susceptible to competing events Diseases and population characteristics
Atrial fibrillation; cardiac failure; coronary heart disease; stroke; aneurysm; prostate cancer; colorectal cancer; breast cancer; chronic leukaemia; cancer screening; critical care; transplant; chronic obstructive pulmonary disease; elderly patients (aged 65+ years).
The full-text articles of all prediction model studies examined within the selected systematic reviews were obtained. Prediction model studies were only considered suitable for inclusion to this review if they met the following criteria:
1. Described the development of a prediction model; and
2. Applied time-to-event analysis methods to develop the prediction model; and 3. Had a clinical setting which reflects at least one of the disease and population
These criteria led to the exclusion of prediction model studies which:
1. Only identified individual prognostic factors without combining to develop a prediction model; or
2. Only validated existing prediction models without developing new models; or 3. Only developed diagnostic, and not prediction, models; or
4. Only used analysis methods other than time-to-event regression, to develop prediction models; or
5. Had a clinical setting not considered to be susceptible to competing events. Conference abstracts were excluded as they contained insufficient detail of the prediction model study for meaningful review.
3.2.2 Data extraction
Relevant information from the included prediction model studies was extracted from the published articles and compiled by the first reviewer (LT) using a data extraction form (Appendix III). A second reviewer (KS) independently extracted information from a third of the included prediction model studies. Any discrepancies between the data extracted by the two reviewers were resolved by discussion between the reviewers (LT and KS). Information was extracted and then narratively summarised in regards to four key items, as outlined below:
3.2.2.1 Item 1: What were the characteristics of each prediction model study?
Information regarding the number of individual prediction models developed within each prediction model study was recorded, as well as the source of the study data (e.g. randomised trial, cohort, or nested case-control) and the total number of participants included in the study.
The reported characteristics of each individual prediction model developed within each prognostic model study were examined using the same criteria and classification system as developed in Chapter 2. This classification system evaluates the prediction model outcome, baseline population, and prediction horizon in turn and combines the results using the process summarised in Figure 2.1. Again, where specific prediction horizons were not reported, the maximum reported follow-up time was recorded as an alternative.
A list of potential competing events likely to prevent the prediction model outcome from occurring was compiled for each prediction model study. Mortality was considered a potential competing event for all individual prediction models predicting outcomes other than all-cause mortality. Potential competing events were additionally determined through examination of the published prediction model study articles for the mention of any events likely to prevent the prediction model events. Further, potential competing events listed for individual prediction models with similar outcomes were compared and added to.
3.2.2.3 Item 3: Were competing events reported in the published prediction model study articles?
The reporting of competing events was examined for the individual prediction models considered to have potential for competing risks bias (classified as low, moderate, or high in Item 2). Information was extracted on the following:
1. The number of prediction model events; 2. The number of reported competing events;
3. The proportion of all reported events which were competing events;
4. The presence of key terms related to competing events (as listed in Table 2.2); 5. Whether Kaplan-Meier curves are presented or discussed;
6. The number of prognostic factors included in the final prediction models; and 7. Any prognostic factors considered to be associated with the competing events.
It has been demonstrated that the level of competing risks bias is strongly associated with the proportion of all observed outcomes that are competing events (Berry et al., 2010, Schumacher et al., 2016, van Walraven and Hawken, 2016, Wolkewitz et al., 2014). Thus this information was extracted. A search of key terms related to competing events (Table 2.2) was conducted using the Adobe Acrobat Reader DC Find function. References to, and depictions of, Kaplan-Meier curves were investigated, as these estimates of absolute risk over time are known to be inflated when competing events are present. Finally, the number of prognostic factors included in the final models, as well as whether these are likely to be associated with competing events, were considered. It has been demonstrated that associations between predictors (such as prognostic factors) and outcomes can change importantly when competing events are appropriately accounted for (Berry et al., 2010, Dignam et al., 2012, Schatzkin and Slud, 1989, Wolkewitz et al., 2014). Appropriately accounting for competing events can alter the magnitude, and in some cases the direction, of the estimated association, particularly when the predictors are strongly associated with the competing event (Berry et al., 2010, Schatzkin and Slud, 1989). Prognostic factors shown to be associated with mortality (anticipated to be a common potential competing event) include age and numerous comorbidities; defined in this instance as chronic diseases or disorders listed in the Charlson comorbidity index (Charlson et al., 1987),
Table 3.2: Comorbidities listed in the Charlson comorbidity index Conditions included in Charlson comorbidity index
Myocaridal infarct, Congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, connective tissue disorder, ulcer disease, mild liver disease, diabetes, hemiplegia, moderate or severe renal disease, diabetes with end organ damage, any tumour, leukemia, lymphoma, modersate or severe liver disease,
matastatic solid tumor, AIDS.
3.2.2.4 Item 4: How were competing events managed in each prediction model study?
Information regarding how competing events were managed within each prediction model study was examined. Specifically:
1. Whether statistical regression methods which appropriately account for competing events, were employed to develop the individual prediction models;
2. Whether participants who experienced competing events were excluded from the study;
3. Whether participants who experienced competing events were censored at the point of experiencing the competing events;
4. whether participants who experienced competing events were managed in any other way;
5. Whether the study also included the validation of the individual prediction models; and
6. How competing events were managed during the validation process. 3.2.3 Analysis methods
For this review, a narrative synthesis of the information extracted from the prediction model studies was conducted. The preliminary synthesis consisted of tabulation and textual descriptions of the extracted information.