There are several other issues especially relevant to discuss when systematically synthesizing evidence on harms. This includes, combining studies only when they are similar enough to warrant combining particularly when evaluating rare and uncommon events, exploring potential sources of heterogeneity in meta- analysis, and adequately considering outcome reporting bias and tools for assessing risk of bias.
1.4.2.1 Rare events
Evaluating comparative risks of uncommon or rare events in systematic reviews can be particularly challenging. A frequent problem in RCTs and systematic reviews is interpreting a non-significant probability value as indicating non- significant difference in risk for a rare AE, particularly when the confidence intervals (CIs) are wide and encompass the possibility of clinically important risks.
For example, in one trial [37] investigating patients with meningitis, “treatment with dexamethasone did not result in an increased risk of AEs” compared with placebo for treatment of hyperglycemia, herpes zoster, or fungal infection because the p-values were greater than 0.20. However, the 95% CIs for relative risk estimates of these three AEs showed clinically significant increase risks. In such as case, researchers should acknowledge the lack of statistical power to assess risks adequately and should interpret the CIs.
19
1.4.2.2 Meta-Analysis
The exact choice of statistical methods to evaluate harms data in a systematic review will depend upon the individual context. Meta-analysis is the preferred method to synthesize evidence in a comprehensive, transparent, and reproducible manner. Though, the rarity of some serious harm outcomes, the relatively small size of some trials, and the restricted patient populations may limit the detection and full evolution of the harms of drugs in individual trials.
The assessment of statistical heterogeneity is appropriate but of lesser concern when dealing with rare but serious AEs where the primary focus is detecting the harm. Commonly employed tests for statistical heterogeneity include; Cochran’s test which is considered relatively underpowered; the Peto odds ratio (OR) method with 95% CI which may provide the best CI coverage, and is more powerful and relatively less bias than random effects analysis when dealing with low event rates; and the fixed effect Mantel-Haenszel test and odds ratio which can be used to reduce confounding, and can adequately deal with zero events within the analysis [36].
1.4.2.3 Outcome Reporting Bias
Furthermore, the credibility of findings from individual trials and from summaries of trials examining a similar research question (that is, systematic reviews and meta-analyses) has been undermined by numerous reporting biases in the published medical literature. Reporting biases are often difficult to detect, but have the potential to discredit earnest efforts towards evidence-based decision making [13, 14, 38].
20
One of the major biases often involved when performing systematic reviews is outcome reporting bias (ORB), which refers to the selective reporting of some results but not others in trial publications. ORB acts in addition to, and in the same direction as “publication bias” of entire studies to produce inflated estimates of treatment effect. The suppression of non-significant findings could lead to the use of harmful interventions.
In a recent study [14] to determine the extent and nature of selective non- reporting of harm outcomes in a cohort study, including 92 systematic reviews of RCTs and non-RCTs, found significantly high evidence of ORB as a result of partially missing reported harms. The study proposes a classification system considering selective outcome reporting that should be appraised outside of the Cochrane risk of bias tool, which is currently being updated. The recommendations from this study are for improvements of reporting harms in both primary studies and systematic reviews.
To overcome ORB, reviewers should also attempt to identify further data from multiple sources including CSRs and clinical trial results registries like the clinicaltrials.gov, as key harms information may be missing from the published trial report.
1.4.2.4 Assessing risk of bias
The development of instruments for assessing risk of bias specifically in studies of harms is still in an early stage of development. General tools for assessing methodological quality can be used but with caution, because they may apply only to the primary focus of the study – usually the beneficial effects of the
21
intervention. For example, for current risk of bias tools like the McMaster Quality Assessment Scale of Harms (also known as McHarm), are designed to detect inflated treatment differences (type I error, i.e., finding of a harm that is not truly present) [39]. The McHarm tool was developed from quality rating of 15 items generated by a Delphi census review of the literature on harms and from previous quality assessment instruments. The subsequent list of the 15 quality criteria was tested for reliability and face, construct, and criterion validity. The McHarm tool is intended for use in conjunction with standardized quality-assessment tools for design-specific internal validity issues.
However due to poor monitoring, lack of clear case definitions and missing data mean that genuine adverse reactions may go undetected or be misclassified. It is therefore believed that systematic reviews of harm should explicitly assess the risk of bias toward the null (e.g., with more attention on harms with lower estimates of risk, like with rare or unexpected events) to prevent a false sense of security (type II error), whereby a drug is erroneously declared safe or not significantly different from the placebo or comparator [40].
The Cochrane handbook for systematic reviews of interventions [41] also highlights some areas of special concern: methods for monitoring and detecting harms, conflicting interests, selective outcome reporting (section 1.4.2.3) and blinding. Furthermore, the Cochrane risk of bias tool for non-randomised studies of interventions (ACROBAT-NRSI) was recently developed allowing for assessments of harms or benefits of an intervention. 1.4.3 Guidance to conducting systematic reviews of harms
22
Studies in the past have also identified other major challenges when developing systematic reviews of harms. This includes a poor quality of information on harms reported in original studies [9, 10, 23, 42], difficulties in identifying relevant studies on harms when using standard systematic search techniques [43, 44], and the lack of a specific guideline to perform a systematic review of harms.
To overcome some of these challenges a number of efforts have been made by collaborative groups and researchers by developing a logical framework and reporting guidelines to guide systematic reviewers.
1.4.3.1 The Cochrane Adverse Effects Methods Group
In 1993 the Cochrane collaboration [45] was formed to organize medical research information in a systematic way to facilitate the choices that health professionals, patients, policy makers and others face in health interventions according to the principles of evidence-based medicine. The group conducts systematic reviews of RCTs which it publishes in the Cochrane library. A few reviews have also studied the results of non-randomised, observational studies.
The Cochrane Adverse Effects Methods Group (AEMG) was formally registered with the Cochrane Collaboration on the 14th June 2007 [46]. The AEMG aims to develop the methods for producing high quality systematic reviews and to advise the Cochrane Collaboration on how the validity and precision of systematic reviews can be improved. A recent publication from the group has provided technical advice for a structured approach to conducting systematic reviews of harms [47], where reviewers are also given general guidance on the
23
assessment of study bias, data collection, analysis, presentation and interpretation of harms in a systematic review. This work will be discussed in more detail in later chapters. The group has also developed and proposed search strategies with appropriate search filters to help identify information on harms [43]. These search strategies aim to help balance the sensitivity (the ability to identify as many relevant articles as possible) with precision (the ability to exclude as many irrelevant articles as possible) when searching bibliographic databases. The AEMG also contribute chapter 14 (Adverse effects) to the Cochrane handbook for systematic reviews of interventions [41].
1.4.3.2 PRISMA Harms Guideline
Additional to the work carried out by the Cochrane AEMGs, in 2009 the Preferred Reporting Items for systematic Reviews and Meta-Analysis (PRISMA) statement [48] was developed as a revision of the Quality of Reporting of Meta- Analysis (QUOROM) statement [49]. The PRISMA statement was developed to guide researchers when conducting systematic reviews and performing meta- analysis in systematic reviews. The statement thus far has mainly focused on efficacy and not on harms. However, in a recent study [50] the quality of reporting in systematic reviews of harms were assessed using their own set of proposed items. The aims of this study were to provide valuable research in the first step of the development for the PRISMA harms extension.