The ongoing frustration with research quality, especially as it relates to the conduct and interpretation of statistical analyses, is well summarised in the opening lines from “Statistical
tests, P values, confidence intervals, and power: a guide to misinterpretations” (2016)294 by the
prominent statisticians and epidemiologists Sander Greenland, Stephen Senn, Ken Rothman, John Carlin, Charles Poole, Steve Goodman, and Doug Altman:
4.2 Evidence of bias
Misinterpretation and abuse of statistical tests … remain rampant. … correct use and interpretation … seems to tax the patience of working scientists. This high cognitive demand has led to … interpretations that are simply wrong … yet these misinterpretations dominate much of the scientific literature.
This article coincided with the unusual step taken by the American Statistical Association of releasing a “Statement on Statistical Significance and P-Values”,383 a response to the
increasing concerns expressed in the literature over recent years about a “reproducibility crisis”384 in all areas of science, including health research.385 And one of the main concerns is
the continuing oversimplification of scientific reasoning encouraged by the use of “null- hypothesis significance testing”, where the standard binary cutoff of p < 0.05 is used to decide whether an effect might be real or not. In terms of causal inference, it can:
• lead to confounders being dropped from models, such as with stepwise regression;128
• encourage the perception by many researchers, including statisticians, that a single study can tell us whether an effect is real or not386
• strengthens the natural human tendency toward overconfidence in the accuracy of our inferences313
Compared to articles criticising the use of null-hypothesis significance testing, very few have been published defending the practice,149 although it may have limited utility for some
research tasks.387
To a large extent, the above article on misinterpretation and misuse of statistics, mirrors those that have appeared regularly for decades. A small sample of titles can be seen in Table 4.1. These commentaries, and the many others that have been published, all suggest that a sizable proportion of health intervention research studies have been analysed and
interpreted poorly, greatly increasing the chance that the results are biased.
Further evidence comes from reviews investigating conflicting results in health research (Table 4.2). When results from difference studies conflict, it suggests that at least one of the studies must be biased.
Randomised controlled trials are also susceptible to bias, though not to confounding by indication if the randomisation was done properly and concealed before allocation. Some articles that found evidence of bias in RCTs are listed in Table 4.3.
Table 4.1 Articles criticising the misuse of statistics from each decade of the last 80 years
Year Article title
1942 “Tests of Significance Considered as Evidence”289
1959 “Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance - Or Vice Versa”388
1960 “The Fallacy of the Null-Hypothesis Significance Test”389
1966 “Statistical Evaluation of Medical Journal Manuscripts”290
1979 “Some Problems of Statistics and Everyday Life”295
1982 “Statistics in Medical Journals”390
1985 “The Religion of Statistics as Practiced in Medical Journals”291
1990 “How Trustworthy is Epidemiologic Research?”391
1994 “The Scandal of Poor Medical Research”292
2005 “Why Most Published Research Findings Are False”392
2018 “Medical Research - Still a Scandal”393
Table 4.2 Reviews investigating conflicting results in health research
Year Article title
2005 “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research”293
2007 “How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis”394
2011 “The Frequency of Medical Reversal”395
2013 “Pioglitazone and Bladder Cancer: Two Studies, Same Database, Two Answers”396
2013 “A Decade of Reversal: An Analysis of 146 Contradicted Medical Practices”397
2015 “Eggs and Beyond: Is Dietary Cholesterol No Longer Important?”398
2016 “A Corpus of Potentially Contradictory Research Claims from Cardiovascular Research Abstracts”399
2018 “Association Between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study”400
4.2 Evidence of bias
Table 4.3 Articles with evidence of bias in randomised controlled trials
Year Article title
1995 “Empirical Evidence of Bias: Dimensions of Methodological Quality Associated with Estimates of Treatment Effects in Controlled Trials”401
2005 “Identifying Outcome Reporting Bias in Randomised Trials on PubMed: Review of Publications and Survey of Authors”402
2008 “Empirical Evidence of Bias in Treatment Effect Estimates in Controlled Trials with Different Interventions and Outcomes: Meta-Epidemiological Study”403
2012 “Observer Bias in Randomised Clinical Trials with Binary Outcomes: Systematic Review of Trials with Both Blinded and Non-Blinded Outcome Assessors”404
2013 “Volunteer Bias in Recruitment, Retention, and Blood Sample Donation in a Randomised Controlled Trial Involving Mothers and Their Children at Six Months and Two Years: A Longitudinal Analysis”405
2014 “Bias Due to Lack of Patient Blinding in Clinical Trials. A Systematic Review of Trials Randomizing Patients to Blind and Non-Blind Sub-Studies”406
2014 “Comparison of Anticipated and Actual Control Group Outcomes in Randomised Trials in Paediatric Oncology Provides Evidence that Historically Controlled Studies are Biased in Favour of the Novel Treatment”407
2015 “Data Interpretation in Analgesic Clinical Trials with Statistically Nonsignificant Primary Analyses: An ACTTION Systematic Review”408
2016 “Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Meta-Epidemiological Studies”409
2017 “Congruence Between Patient Characteristics and Interventions May Partly Explain
Medication Adherence Intervention Effectiveness: An Analysis of 190 Randomized Controlled Trials from a Cochrane Systematic Review”410
2017 “Cherry-Picking by Trialists and Meta-Analysts Can Drive Conclusions about Intervention Efficacy”411
2017 “Simple Randomization Did Not Protect Against Bias in Smaller Trials”412
2018 “A Review of Cluster Randomized Trials Found Statistical Evidence of Selection Bias”413
Finally, evidence of bias is also suggested by articles (Table 4.4) identifying problems with methodologies, errors, reporting biases, and also by retractions, where the implication is that many more articles containing errors or poor judgement in methodology, as well as
deliberate fraud, would be retracted if those problems were discovered.317
,414
Table 4.4 More articles with evidence of bias from the last 5 years
Year Article title
2013 “Why Has the Number of Scientific Retractions Increased?”416
2015 “Biased and Inadequate Citation of Prior Research in Reports of Cardiovascular Trials is a Continuing Source of Waste in Research”417
2017 “Indirect Evidence of Reporting Biases was Found in a Survey of Medical Research Studies”418
2017 “Top Ten Errors of Statistical Analysis in Observational Studies for Cancer Research”419
2017 “The Distribution of P-Values in Medical Research Articles Suggested Selective Reporting Associated with Statistical Significance”420
2017 “Survival Biases Lead to Flawed Conclusions in Observational Treatment Studies of Influenza Patients”421
2018 “High and Unclear Risk of Bias Assessments are Predominant in Diagnostic Accuracy Studies Included in Cochrane Reviews”422
2018 “Interpretation of Epidemiologic Studies Very Often Lacked Adequate Consideration of Confounding”423
2018 “Kaplan-Meier Survival Analysis Overestimates Cumulative Incidence of Health- Related Events in Competing Risk Settings: A Meta-Analysis”424
2018 “Three Risk of Bias Tools Lead to Opposite Conclusions in Observational Research Synthesis”425