Bias can have a negative impact on the evidence produced by systematic reviews, because there is a tendency to over inflate studies with more positive results, even though the study quality might be poor. Moreover, this focus on positive over negative estimates of effects can be detrimental as sometimes false or weak positive results might lead to the promotion of
69
inappropriate interventions or practices. For this reason it is important that systematic reviews are truly systematic and as such attempt to include all the possible evidence available and always report any potential for bias in the included studies or in the review itself.
According to Song et al. (2010) and Stern et al. (2011) there are a few strategies that can help reduce literature bias in systematic reviews, these include the inclusion of unpublished literature, the use of trial registers, the search strategy, the assessment of the quality of the included studies and also of the systematic review itself.
Systematic reviews normally include only studies that have been published and are available through mainstream journals however, unpublished studies may also add to the body of knowledge and help avoid or decrease the risk of bias. For example, in a systematic review Hopewell et al. (2007b) investigated the impact of grey literature in
randomised controlled trials of healthcare interventions, and found that of the five studies included, all showed that published trials had a general greater treatment effect (9%) than the trials reported within the grey literature. The authors also found that meta-analyses included more published trials than those existing within the grey literature as these were mainly abstracts (55%) and unpublished data (30%). The authors
conclude that evidence is limited to suggest that grey literature trials are of poorer methodological quality than published trials.
According to Stern et al. (2011) one of the main reasons why grey
literature is not included in systematic reviews is due to the fact that they are not peer-reviewed. However as the peer-review process in itself can be biased, as such it should not be considered as the ultimate test of quality.
Another strategy to try and avoid or reduce bias in systematic reviews relates to the use of trial registers (Chan et al. 2004a, Stern et al. 2011). Trial registers involve registering studies from inception, including
70
transparency can be assured. When it comes to the review process and for publishing the final study, outcome measures can be compared with the initial protocol. If discrepancies are found researchers can then justify those discrepancies enhancing transparency and thus reducing bias. Song et al. (2010) suggest another method to help avoid or reduce bias involves the search strategy. For a systematic review to be rigorous, the search strategy needs to be well planned in order to retrieve all the
relevant studies related to the study subject area. For this to happen there has to be a balance between sensitivity, which is the identification of the relevant studies within the total number of studies that exist, and precision, which is the ability to identify the relevant number of studies within those that have been retrieved (Song et al. 2010). It is also necessary that the search terms used are not too specific as this will reduce the sensitivity of the search. Instead more generic terms, whenever possible, should be used (Song et al. 2010). Furthermore, it is important that reviewers use a varied range of search modalities that spans from electronic databases, to hand searching of journals, citation tracking and contacting of experts in the area, as this shows that the reviewer has extensively searched for all the available literature on the topic being investigating and thus tried to reduce the chances of bias. Moreover it is important that researches avoid language restrictions as this will limit the available evidence.
To avoid, reduce or even detect literature bias in systematic reviews, it is essential that the studies included and the outcomes of the systematic reviews are reported and its methodology is sound. These elements will invariable influence the quality of the systematic reviews. For this purpose tools and guidelines have been developed to help researchers report individual trials and also to assess the quality of systematic reviews.
In terms of assessing individual studies and thus improving their reporting, the CONSORT statement and the STROBE statement have been
developed, the former focusing on randomised controlled trials and the latter on observational studies in epidemiology. The CONSORT statement is a 25 item checklist that analysis if all the steps in the research process
71
have been reported (see Appendix 1) and provides guidance for reporting randomised controlled trials, through the use of a checklist of items that should be included in reports of randomised controlled trials (Moher et al. 2010). Through the use of CONSORT researchers will be able to develop and report evidence in a more clear, complete and transparent way as well as being able to appraise the quality of the evidence produced (Schulz et
al. 2010). One can conclude that if researchers use this tool consistently
their study will be of high quality and the evidence produced will accurately inform practice.
The Strengthening the Reporting of Observational Studies in Epidemiology – STROBE statement, is a guideline that gives recommendations on what should be included in observational studies so that these are accurate and complete (von Elm et al. 2007). This statement is based on the premise that all research should be transparent, thus reducing bias, so that those accessing the evidence can be sure that this is of high quality. Like the CONSORT statement, STROBE statement is a checklist, in this case of 22 items (see Appendix 2), that are considered by its authors to be of
fundamental importance for good reporting of observational studies (von Elm et al. 2007). The STROBE statement also analysis if all the steps of the research process have been followed. Although the purpose of STROBE is not to assess quality of published observational research, if the reporting of observational studies is properly done and bias avoided at the different stages, then one can argue that this will contribute to a higher quality study than if the guidelines were not followed.
In relation to systematic reviews the PRISMA statement, the AMSTAR tool and the STAGE guidelines are used to help improve the reporting of
systematic reviews and assess their quality, respectively.
The Preferred Reporting Items for Systematic Reviews and Meta-analysis – PRISMA statement, previously known as the QUOROM statement, was developed to help researchers improve reporting of systematic reviews and meta-analyses, however it can also be used for reporting systematic
72
reviews that included other type of studies (Moher et al. 2009). PRISMA comprises of a 27 item checklist (see Appendix 3), where the items that need to be included when reporting a systematic review are identified, and it also comprises of a flow-diagram that gives information regarding the search strategy used in the systematic review (Moher et al. 2009). Though its main purpose is to help improve reporting of systematic reviews and meta-analyses and it is not a quality assessment tool per se, the PRISMA statement can also be used for critical appraisal of systematic reviews, because if there are inconsistencies in the reporting, this could mean that the research process and respective steps were not rigorously followed which in turn can lead to biased conclusions and evidence that will downstage the quality of evidence produced.
In relation to the AMSTAR tool, its focus is to assess the methodological quality of systematic reviews (Shea et al. 2007, Shea et al. 2009). This tool consists of an 11 items (see Appendix 4) checklist and allows researchers to assess if their systematic review has methodological quality. If the quality is found to be low, researchers can address the problems identified and improve their study and consequently the
evidence produced, thus consciously contributing for the dissemination of high quality evidence.
The final tool/guideline that will help improve the quality of systematic reviews is the STAGE guidelines. The STAGE guidelines are a system that rates the quality of evidence and stages the strength of
recommendations from systematic reviews, health technology assessments and clinical practice guidelines (Guyatt et al. 2011b). Focusing on systematic reviews, STAGE tries to outline the quality of evidence produced by systematic reviews, looking at the extent to which there can be confidence that an estimate effect is appropriate
(Schünemann et al. 2011). In order to reach transparency STAGE rates the quality of evidence in four levels: high, moderate, low and very low (Guyatt et al. 2008). These guidelines will help assess if the quality of evidence produced by systematic reviews is of high quality looking to
73
reduce risk of bias, risk of publication bias, imprecision, indirectness and inconsistency, that have been previously discussed (Schünemann et al. 2011).
In summary, researchers have a variety of tools that can be used to
improve not only reporting of individual studies and systematic reviews but also the quality of the study and the evidence produced, by identifying and avoiding/reducing literature bias. This will invariable help produce high quality and best evidence that can be used to inform and improve practice.
2.4. Systematic reviewing of quantitative or qualitative research