Elementos gramaticales 105

While systematic reviews have generally focussed on the evaluation of the accuracy of a single test, reviews comparing the accuracy of two or more tests are increasingly being published. For comparing health care interventions, properly conducted RCTs are regarded as the most valid source of evidence of comparative effectiveness, although in their absence studies with non-randomized designs may also be considered.82-85 Both RCTs and systematic reviews of RCTs are available to guide intervention selection for many conditions. However, clinical investigators and funders do not seem to have demanded similarly rigorous standards in the creation of reliable evidence for selecting diagnostic tests.

Systematic reviews of comparative accuracy often undertake separate meta-analyses for each test, and then compare their results as shown in Figure 1.11 (panel A). Comparing separate meta-analyses of each test can be likened to making comparisons of single arms of RCTs of interventions, or between case series.86 This indirect between-study (uncontrolled) test comparison uses a different set of studies for each test and so does not ensure like-with-like comparisons; the difference in accuracy is prone to confounding due to differences in patient groups and study methods.

45

Figure 1.11| Approaches for comparing test accuracy

Systematic reviews of comparative accuracy often undertake separate meta-analyses for each test and then compare their results as shown in (A) using one study for each test. The second figure (B) shows a direct comparison using a comparative study (comparison within patients or between randomized groups).

Although direct comparisons based on only comparative studies are likely to ensure an unbiased comparison and enhance validity (Figure 1.11 panel B), such analyses may not always be feasible due to limited availability of comparative studies. 71,87 For example, four modalities—duplex ultrasonography (DUS), computed tomography angiography (CTA), magnetic resonance angiography (MRA), and digital subtraction angiography (DSA)—are used to image steno-occlusions in lower limb peripheral arterial disease (PAD). In routine clinical practice, DSA is not used as an alternative to DUS, CTA or MRA, but is the reference standard used to determine the diagnostic accuracy of the other three modalities. For certain clinical questions, a reference standard may also be the alternative test used in practice, i.e., also a comparator test. Consequently, the reference standard is sometimes incorrectly referred to as the comparator.

(A) Indirect (between-study) comparison (B) Direct (head-to-head) comparison

Series of patients

Index test A

Reference standard

Estimate sensitivity and specificity for

test A

Compare test accuracy using different sets of

studies

Series of patients

Index test B

Reference standard

Estimate sensitivity and specificity for

test B

Series of patients

Index test A

Reference standard

Estimate sensitivity and specificity for

test A

Compare test accuracy using same set of

studies

Index test B

Estimate sensitivity and specificity for

46 DUS, CTA and MRA have specific benefits and limitations but the clinical indications for CTA are very similar to those of MRA.88 Nonetheless, no study has directly compared the diagnostic accuracy of MRA and CTA.89 In a systematic review of the diagnostic accuracy of CTA, MRA and DUS that included 58 studies, Collins et al90 only found two studies that compared MRA and DUS in the same study population. One of the two studies performed all tests in all patients while the other study only performed MRA in a subset of patients. The authors concluded that MRA has better overall diagnostic accuracy than CTA or DUS.

An indirect comparison uses all eligible studies that have evaluated at least one of the tests of interest thus maximizing use of the available data (Figure 1.12). Therefore, an indirect test comparison can be regarded as a mixed test comparison when both comparative and non- comparative studies of one or more tests are included (i.e. direct and indirect comparisons are combined). However, a distinction is not usually made between both approaches and they are simply termed indirect comparisons. This is unlike indirect comparisons and mixed treatment comparisons (or network meta-analyses) of interventions where there is a clear distinction between the two methods of obtaining indirect evidence of treatment effectiveness.91-93 Furthermore, unlike indirect comparisons of interventions, these naïve indirect test

comparisons do not use a common control to adjust for differences in average test positivity (outcome) rates between studies.

47

Figure 1.12| Types of test comparisons in a comparative accuracy meta-analysis

For simplicity, a pairwise comparison is shown but a test comparison may include more than two tests. The mixed test comparison includes head-to-head studies of both index tests as well as evaluations of either of the two tests. The mixed test comparison is also an indirect

comparison.

The high degree of heterogeneity in estimates of sensitivity and specificity commonly

observed in meta-analyses of test accuracy94 raises concern that decisions based on the results of comparisons made between meta-analyses may be unreliable because the likelihood of confounding is high. The likely magnitude of any bias is unknown, but if substantial, robustly designed comparative test accuracy studies (see Figure 1.10) should be more routinely

undertaken and preferred for evidence to guide test selection. Therefore, the availability of comparative studies and the feasibility of direct comparisons warrant investigation given the

48 potential value of comparative accuracy reviews as a surrogate in the absence of direct

evidence about test effectiveness.

There is also concern that suboptimal methods are being used to estimate test accuracy and to make conclusions about the superiority of one test compared to another. Hierarchical meta- analytic methods have been increasingly adopted over time, but their use still remains limited.27 In contrast to meta-analyses of interventions, recommended methods may sometimes be impractical due to the limited number of included studies compared to the complexity of the methods. It is unclear whether simpler methods can be an appropriate alternative in such situations. Therefore meta-analytic methods for valid comparisons of tests are needed to avoid misleading conclusions and recommendations.