Original articleCitation bias of hepato-biliary randomized clinical trials
Introduction
Metaanalyses are increasingly being recognized as a reliable tool for summarizing research evidence [1], but the results can be misleading if based on a biased sample of the available evidence [2]. For example, metaanalyses will provide an exaggerated estimate of the effect of an intervention if omitting negative trials (i.e., trials showing no significant effect). There are several ways by which such bias can be introduced [3]. First, positive studies (i.e., studies with statistically significant results) are more likely to be published and have a significantly shorter time to publication than negative trials 4, 5. Second, among published studies, those with positive results are more likely to be published in English [6], to be published in journals indexed in electronic databases [3], and to be published several times [4].
Bias in the identification and inclusion of trials for metaanalyses may also be affected by the number of times they are cited [7]. However, the evidence concerning the extent and direction of citation bias is equivocal. Two studies found that positive trials (i.e., trials showing a significant intervention benefit) are cited more often than negative trials 8, 9, 10. However, two other studies showed the direct opposite (i.e., that negative trials are cited more often than positive trials) 11, 12. The discrepancy between these findings may reflect different citation habits in different areas [13], or confounding (e.g., language bias [6], database bias [3], or variations in the methodological quality 14, 15, 16). In the present study, we assessed the extent and direction of citation bias among trials on several hepato-biliary diseases published in English-language journals indexed in MEDLINE.
Section snippets
Selection of studies
We included all randomized trials from the following arbitrarily selected general and specialist English-language journals that were indexed in MEDLINE and were likely to publish hepato-biliary randomized clinical trials: The New England Journal of Medicine, Annals of Internal Medicine, Lancet, Gastroenterology, Hepatology, Journal of Hepatology, Digestive Diseases and Sciences, Digestion, Liver, Scandinavian Journal of Gastroenterology, and Journal of Clinical Gastroenterology. Trials were
Description of trials
In total, 530 trials (available on request) fulfilled the inclusion and none of the exclusion criteria. According to the Science Citation Index, the trials received a total of 2,327 citations during the first two calendar years after the year of publication. The median citation frequency of all trials was 2 (interquartile range, 0–6; range, 0–50). One hundred sixty trials (30%) were not cited. The study outcome was classified as positive in 374 trials (71%). The trials covered 12 disease areas
Discussion
In the present study, we found significant citation bias of hepato-biliary randomized clinical trials. Trials with a positive outcome were cited significantly more often than trials with a negative outcome. The disease area and the generation of the allocation sequence were significant independent predictors of the citation frequency but did not explain the association between the study outcome and citation frequency.
The present study includes trials from several disease areas within one
Acknowledgments
We thank Peter C. Gøtzsche, Thorkild I.A. Sørensen, and two peer referees at Journal of Clinical Epidemiology for valuable comments to an earlier draft of this manuscript. We also thank Ninna Frydendal, Dimitrinka Nikolova, Sarah L. Frederiksen, and Nader Salasshahri for their help with the identification and retrieval of trials. This work was supported by grants from The Danish Medical Research Council and The 1991 Pharmacy Foundation in Denmark.
References (25)
“Negative trials” are positive!
J Hepatol
(1998)- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Language bias in randomised controlled trials published in English and German
Lancet
(1997) Quotation bias in reviews of the diet-heart idea
J Clin Epidemiol
(1995)- et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
(1998) - et al.
Quality assessment of reports on clinical trials in the Journal of Hepatology
J Hepatol
(1998) - et al.
How consumers and policymakers can use systematic reviews for decision making
Ann Intern Med
(1997) - et al.
Bias in location and selection of studies
BMJ
(1998) - et al.
Publication biasevidence of delayed publication in a cohort study of clinical research projects
BMJ
(1997) - et al.
Publication and related biases
Health Technol Assess
(2000)
Reference bias in reports of drug trials
BMJ
Cholesterol lowering trials in coronary heart diseasefrequency of citation and outcome
BMJ
Cited by (60)
Citation bias in the literature on dietary trans fatty acids and serum cholesterol
2019, Journal of Clinical EpidemiologyScientific citations favor positive results: a systematic review and meta-analysis
2017, Journal of Clinical EpidemiologyCitation Excerpt :The direction of citation bias was fairly consistent: with some exceptions [6,7], most studies reported that positive articles were cited more often than negative articles (Table 1). Most of the studies are biomedical [7–42], but some also concern the social [43–49] and natural sciences [6,50,51], or a combination of these [52]. The biomedical studies ranged from highly specific fields—such as the relationship between job strain and cardiovascular disease [16], or the treatment of chronic nonspecific lower-back pain [8]—to broader categories such as cardiovascular medicine [10].
Brief alcohol intervention trials conducted by higher prestige authors and published in higher impact factor journals are cited more frequently
2016, Journal of Clinical EpidemiologyCitation Excerpt :The prevailing hypothesis is that methodologically sound publications will be cited more frequently than lower quality publications. This hypothesis is unsubstantiated, however, given that citation rates have been linked to other factors unassociated with study quality, including the direction and statistical significance of effects [1−5]. Two other factors that may contribute to citation rates above and beyond study quality, however, are author prestige and journal impact factors.
Citation networks of related trials are often disconnected: Implications for bidirectional citation searches
2014, Journal of Clinical EpidemiologyCitation Excerpt :The few trials that are selected to cite are likely selected in a biased manner. Trials are more likely to be cited if they report statistically significant results [7–9], if the authors are from the same country [10], and other reasons unrelated to the strength of evidence available from the prior trial [11–16]. Despite limited citation of prior trials, checking references of eligible studies does lead to the identification of additional relevant studies for systematic reviews.
Reducing waste from incomplete or unusable reports of biomedical research
2014, The LancetCitation Excerpt :Such poor citation of relevant research also seems to be selective. For example, an analysis of 530 randomised clinical trials in hepatobiliary diseases reported in 11 journals between 1985 and 1996, showed that positive (statistically significant) studies were more than twice as likely to be cited than were negative (not statistically significant) ones, compounding the problem of publication bias.64 Another analysis of 111 trials in rheumatoid arthritis showed that a higher proportion of the positive trials were cited than were the negative trials for the same question.65