Elementos didácticos 109

1.7.1 Research questions

Given the challenges and uncertainties highlighted above, the primary aim of this thesis is to assess the reliability and transparency of evidence derived from systematic reviews and meta- analyses of comparative accuracy, including the validity of the meta-analytic methods used to synthesise the evidence. To achieve this objective, the following research questions will be addressed:

1. How have comparisons of test accuracy been performed and reported in published systematic reviews and meta-analyses? What are the statistical methods used and are the methods and findings well reported?

49 2. How feasible are direct comparisons (i.e. how often are comparative studies available)?

Are meta-analyses of comparative test accuracy studies more reliable than meta- analyses of non-comparative studies?

3. How should meta-analyses be undertaken to compare test accuracy

a. when both tests are evaluated in different sets of studies (indirect uncontrolled comparisons);

b. when studies directly compare both tests (head-to-head comparisons); c. when there is a mixture of non-comparative and comparative studies (mixed

comparison);

d. when there are more than two tests (multiple test comparison)? Do methods give the same results in each of the situations above?

4. How should meta-analyses be undertaken when there are few studies or sparse data?

1.7.2 Thesis outline

Chapter 2 introduces several examples of published systematic reviews comparing the accuracy of two or more tests. These examples are from some of the reviews the author has worked on in the past five years during the course of this PhD research. This chapter aims to highlight key methodological issues by providing practical demonstrations using real life case studies. The chapter also shows how the problems were addressed in the reviews and

underscores the need for research undertaken in Chapters 4 to 8.

Chapter 3 focuses on the search methods for identifying systematic reviews that assessed the accuracy of at least two tests, and the methods for comparative meta-analysis investigated in the thesis. Thus the aim of this chapter is to collate the data sources and search strategies used

50 to obtain methods and materials for the thesis. The chapter also contains the search results for the primary cohort of reviews from which the subset of reviews used in chapters 4 to 6 were derived by applying eligibility criteria relevant to the objectives of each chapter.

Chapter 4 provides an overview of data synthesis methods in published reviews that assessed at least two tests. The aim of the chapter is to provide a descriptive survey of recent practice, and to identify shortcomings in methods and reporting with a view to making

recommendations for improvements. Recent reviews selected from the primary cohort of reviews identified in Chapter 3 are examined in detail, and their general, methodological and reporting characteristics are summarised. The chapter describes how reviews handled test comparisons in terms of the strategies (indirect and/or direct comparisons) and meta-analytic methods used, presentation of results, and how review findings were interpreted in the context of the strength of the evidence. Examples of good practice are identified to aid future review authors in preparing these reviews.

Chapter 5 describes the assessment of the availability of comparative test accuracy studies, and compares meta-analyses of comparative studies (direct comparisons) and meta-analyses of non-comparative studies (indirect comparisons). This chapter aims to determine the existence and magnitude of discrepancies between meta-analyses of direct and indirect comparisons, and to provide empirical evidence of the importance of comparative accuracy studies. The meta-analyses used in this empirical evaluation are derived from the cohort of reviews identified in Chapter 3.

51 Chapter 6 considers methods for comparative meta-analysis that were identified from searches reported in Chapter 3. Chapter 6 aims to describe the properties of the methods such as model specification, modelling assumptions, and the advantages and limitations of each method. This methodological review will provide an overview of the available comparative meta- analysis methods and will gaps where appropriate methods are lacking.

In Chapter 7, the performance of methods identified in Chapter 6 that are deemed to be methodological rigorous or are frequently used in practice will be empirically evaluated by using a subset of the reviews identified in Chapter 3. The evaluation includes investigating the impact of alternative methods and modelling assumptions on conclusions in order to identify the most appropriate methods and to provide practical guidance for meta-analysts.

Chapter 8 investigates the performance of hierarchical meta-analytic methods in situations with few studies or sparse data for a single index test. This is a common problem faced by meta-analysts and the aim of this chapter is to identify situations where complex hierarchical methods are likely to give model fitting problems and misleading results, and to suggest simpler appropriate meta-analytic methods. The meta-analyses in two reviews are used as motivating examples and simulation is used to generate datasets that reflect realistic scenarios for meta-analyses of test accuracy studies. The performance of seven hierarchical models incorporating increasing simplifications is investigated and the chapter concludes with recommendations for practice.

Although the main emphasis of this thesis is on test comparisons, the simulation study only considers the meta-analysis of a single test. This comprehensive simulation has wide

52 implications and is a necessary preliminary step for establishing the validity of simpler

models in the context of the evaluation of a single test prior to considering applications to test comparisons. This is also important because a comparative meta-analysis is not always possible with few studies of one or more of the tests being compared. As such separate meta- analyses may be required for some tests in addition to the comparative meta-analysis of tests with sufficient data. The chapter includes a section on how the results may be generalised to test comparisons.

Chapter 9 concludes the thesis by summarising the key findings from Chapters 2 to 8. The chapter draws together the various issues addressed in the thesis and seeks to provide a coherent summary of both the problems raised and the recommendations that were developed based on the thesis findings. The chapter also offers suggestions for future research, and the overall limitations and conclusions of the thesis.

53