Original article
Citation bias of hepato-biliary randomized clinical trials

https://doi.org/10.1016/S0895-4356(01)00513-3Get rights and content

Abstract

The objective of this study was to assess whether trials with a positive (i.e., statistically significant) outcome are cited more often than negative trials. We reviewed 530 randomized clinical trials on hepato-biliary diseases published in 11 English-language journals indexed in MEDLINE from 1985–1996. From each trial, we extracted the statistical significance of the primary study outcome (positive or negative), the disease area, and methodological quality (randomization and double blinding). The number of citations during two calendar years after publication was obtained from Science Citation Index. There was a significant positive association between a statistically significant study outcome and the citation frequency (β, 0.55, 95% confidence interval, 0.39–0.72). The disease area and adequate generation of the allocation sequence were also significant predictors of the citation frequency. We concluded that positive trials are cited significantly more often than negative trials. The association was not explained by disease area or methodological quality.

Introduction

Metaanalyses are increasingly being recognized as a reliable tool for summarizing research evidence [1], but the results can be misleading if based on a biased sample of the available evidence [2]. For example, metaanalyses will provide an exaggerated estimate of the effect of an intervention if omitting negative trials (i.e., trials showing no significant effect). There are several ways by which such bias can be introduced [3]. First, positive studies (i.e., studies with statistically significant results) are more likely to be published and have a significantly shorter time to publication than negative trials 4, 5. Second, among published studies, those with positive results are more likely to be published in English [6], to be published in journals indexed in electronic databases [3], and to be published several times [4].

Bias in the identification and inclusion of trials for metaanalyses may also be affected by the number of times they are cited [7]. However, the evidence concerning the extent and direction of citation bias is equivocal. Two studies found that positive trials (i.e., trials showing a significant intervention benefit) are cited more often than negative trials 8, 9, 10. However, two other studies showed the direct opposite (i.e., that negative trials are cited more often than positive trials) 11, 12. The discrepancy between these findings may reflect different citation habits in different areas [13], or confounding (e.g., language bias [6], database bias [3], or variations in the methodological quality 14, 15, 16). In the present study, we assessed the extent and direction of citation bias among trials on several hepato-biliary diseases published in English-language journals indexed in MEDLINE.

Section snippets

Selection of studies

We included all randomized trials from the following arbitrarily selected general and specialist English-language journals that were indexed in MEDLINE and were likely to publish hepato-biliary randomized clinical trials: The New England Journal of Medicine, Annals of Internal Medicine, Lancet, Gastroenterology, Hepatology, Journal of Hepatology, Digestive Diseases and Sciences, Digestion, Liver, Scandinavian Journal of Gastroenterology, and Journal of Clinical Gastroenterology. Trials were

Description of trials

In total, 530 trials (available on request) fulfilled the inclusion and none of the exclusion criteria. According to the Science Citation Index, the trials received a total of 2,327 citations during the first two calendar years after the year of publication. The median citation frequency of all trials was 2 (interquartile range, 0–6; range, 0–50). One hundred sixty trials (30%) were not cited. The study outcome was classified as positive in 374 trials (71%). The trials covered 12 disease areas

Discussion

In the present study, we found significant citation bias of hepato-biliary randomized clinical trials. Trials with a positive outcome were cited significantly more often than trials with a negative outcome. The disease area and the generation of the allocation sequence were significant independent predictors of the citation frequency but did not explain the association between the study outcome and citation frequency.

The present study includes trials from several disease areas within one

Acknowledgments

We thank Peter C. Gøtzsche, Thorkild I.A. Sørensen, and two peer referees at Journal of Clinical Epidemiology for valuable comments to an earlier draft of this manuscript. We also thank Ninna Frydendal, Dimitrinka Nikolova, Sarah L. Frederiksen, and Nader Salasshahri for their help with the identification and retrieval of trials. This work was supported by grants from The Danish Medical Research Council and The 1991 Pharmacy Foundation in Denmark.

References (25)

  • P.C. Gøtzsche

    Reference bias in reports of drug trials

    BMJ

    (1987)
  • U. Ravnskov

    Cholesterol lowering trials in coronary heart diseasefrequency of citation and outcome

    BMJ

    (1992)
  • Cited by (60)

    • Scientific citations favor positive results: a systematic review and meta-analysis

      2017, Journal of Clinical Epidemiology
      Citation Excerpt :

      The direction of citation bias was fairly consistent: with some exceptions [6,7], most studies reported that positive articles were cited more often than negative articles (Table 1). Most of the studies are biomedical [7–42], but some also concern the social [43–49] and natural sciences [6,50,51], or a combination of these [52]. The biomedical studies ranged from highly specific fields—such as the relationship between job strain and cardiovascular disease [16], or the treatment of chronic nonspecific lower-back pain [8]—to broader categories such as cardiovascular medicine [10].

    • Brief alcohol intervention trials conducted by higher prestige authors and published in higher impact factor journals are cited more frequently

      2016, Journal of Clinical Epidemiology
      Citation Excerpt :

      The prevailing hypothesis is that methodologically sound publications will be cited more frequently than lower quality publications. This hypothesis is unsubstantiated, however, given that citation rates have been linked to other factors unassociated with study quality, including the direction and statistical significance of effects [1−5]. Two other factors that may contribute to citation rates above and beyond study quality, however, are author prestige and journal impact factors.

    • Citation networks of related trials are often disconnected: Implications for bidirectional citation searches

      2014, Journal of Clinical Epidemiology
      Citation Excerpt :

      The few trials that are selected to cite are likely selected in a biased manner. Trials are more likely to be cited if they report statistically significant results [7–9], if the authors are from the same country [10], and other reasons unrelated to the strength of evidence available from the prior trial [11–16]. Despite limited citation of prior trials, checking references of eligible studies does lead to the identification of additional relevant studies for systematic reviews.

    • Reducing waste from incomplete or unusable reports of biomedical research

      2014, The Lancet
      Citation Excerpt :

      Such poor citation of relevant research also seems to be selective. For example, an analysis of 530 randomised clinical trials in hepatobiliary diseases reported in 11 journals between 1985 and 1996, showed that positive (statistically significant) studies were more than twice as likely to be cited than were negative (not statistically significant) ones, compounding the problem of publication bias.64 Another analysis of 111 trials in rheumatoid arthritis showed that a higher proportion of the positive trials were cited than were the negative trials for the same question.65

    View all citing articles on Scopus
    View full text