In this chapter, an evaluation of the presence and reporting of competing events in 31 systematic reviews of prediction model studies was conducted. A classification system for the risk of competing risks bias was developed and applied to each systematic review. The key findings and conclusions of this evaluation are summarised in Box 2.1 and discussed below.
Box 2.1: Key findings
1. Competing events are often present in the prediction models contained within systematic reviews of prediction model studies.
2. Systematic reviews of prediction model studies rarely report the presence or assessment of competing risks.
3. A wide variety of quality assessment tools are utilised in systematic reviews of prediction models studies, few consider competing risks.
2.4.1 Key findings
The key findings of this evaluation are presented and discussed in more detail below:
2.4.1.1 Competing events are often present in the prediction models contained within systematic reviews of prediction model studies.
Of the 31 systematic reviews evaluated, 90.3% were found to include prediction models with outcomes other than all-cause mortality, making them susceptible to competing events. Further, 61.3% were classified as having high potential for competing risks bias, when the prediction model outcome, baseline population, and prediction horizon were considered. This suggests that competing events are commonly present in prediction model development studies, and competing risks bias will often be a potential concern for systematic reviews of these studies. Consequently, the competing events should be acknowledged and examined in systematic reviews
2.4.1.2 Systematic reviews of prediction model studies rarely report the presence or assessment of competing risks.
Despite the high potential for competing events in prediction model studies, few systematic reviews reported or assessed issues related to competing risks in the published articles. Only two (6.5%) of the systematic review articles evaluated directly reported on competing risks. If the statistical methods used to develop prediction models are not appropriate for the complexities of the data, the estimated predictive performance of the model may be biased (Moons et al., 2015). Not reporting the presence of competing risks could result in biases in the conclusions made by the systematic reviews.
2.4.1.3 A wide variety of quality assessment tools are utilised in systematic reviews of prediction models studies, but few consider competing risks
This evaluation identified a number of quality assessment tools currently being used to assess prediction model studies in systematic reviews. The array and inconsistency of quality assessment tools used by the included systematic reviews highlights the absence of agreed quality criteria for assessing the risk of bias in prognostic modelling studies. However, the Prediction study Risk Of Bias Assessment Tool (PROBAST) has recently been developed to address this issue. This was the only quality assessment tool used by the systematic reviews which explicitly referred to competing events. The tool directly refers to competing risks when considering the risk of bias in the analysis of the prediction mode study: “Were complexities in the data
(e.g., censoring, competing risks, sampling of control participants) accounted for appropriately?” (Wolff et al., 2019). This tool was released in 2019, and so it hopefully
will help to improve the assessment of competing risks bias in future systematic reviews of prediction model studies.
To the author’s knowledge, this is the largest evaluation of competing events in systematic reviews of prediction model studies. The evaluation highlights the high presence of competing events in prediction model research, motivating the need to consider competing events in systematic reviews which aim to evaluate such studies. Nevertheless, the evaluation does not encompass all systematic reviews of prediction model studies. The systematic reviews which were not considered, due to not meeting inclusion criteria, may differ to those evaluated. For example, inclusion of studies containing the terms “systematic review” or “meta-analysis” in the article title is likely to identify systematic review conforming to the PRISMA reporting guidelines. However, reviews which follow reporting guidelines may be of better quality, or at least report on more key items, than those which do not.
The classification system used to assess the potential for competing risk bias was developed by combining criteria from articles which similarly aimed to assess the presence of competing risks bias in other settings (Austin and Fine, 2017, Koller et al., 2012, Schumacher et al., 2016, Walraven and McAlister, 2016). Criteria thought to be applicable to prediction model studies, and likely to be reported in the systematic review articles, were selected. Criteria discussed in the articles but not included in the classification system include: the reporting of Kaplan-Meier estimates and assessing the number and proportion of competing events. These were not included as criteria in the classification system as it was thought these would not typically be reported in systematic review articles. The tool was developed to pragmatically determine whether competing events were likely to be present in the included prediction model studies, as it was considered beyond the scope of the thesis to obtain individual study data for all included studies to determine whether competing events were truly present. Further, the presence of competing events in a prediction model study does not necessarily imply that the study contains competing risks bias. If a study identifies the competing
regression analysis, then no such bias will be observed. It is suspected by the author that appropriate methods are rarely utilised in prediction model research, and thus the presence of competing events is likely to be indicative of competing risks bias. Thus further research to determine whether competing events were indeed present, and investigating the management of these competing events, is required.
The following chapter describes a review which investigates the presence, reporting, and management of competing events in prediction model studies classified as high potential for competing risks bias, i.e. the most susceptible to competing risks bias.