ArticlesDoes the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?
Introduction
A meta-analysis is multifactorial. Decisions need to be made about how to handle various factors, such as language of publication, quality, and publication status, at the individual study level. Within the domain of publication status, a major factor to consider is the inclusion of grey literature (ie, studies that are unpublished, have limited distribution, and/or are not included in bibliographical retrieval system).1
The inclusion of grey literature in a meta-analysis may help to overcome some of the problems of publication bias, and provide a more complete and objective answer to the question under consideration. However, it has been reported that only 31% of published meta-analyses include grey literature.2 This omission may be because the nature of grey literature makes its exclusion more convenient; it is difficult to retrieve, it is frequently incomplete, and its quality may be difficult to assess. We aim to provide empirical evidence about the impact of the exclusion of grey literature from meta-analyses on the estimate of intervention effectiveness.
Section snippets
Selection of meta-analyses
A sample of 135 meta-analyses were drawn randomly from an existing database of 455 meta-analyses of randomised clinical trials.3, 4 The database was established in 1996 through MEDLINE searches from 1966 to 1995 with a detailed search strategy assembled with the aid of an information specialist.
Eligibility criteria
To be eligible, a study has to be deemed a meta-analysis (included pooled analyses of the results of several independent primary studies), the associated studies had to be identifiable, and at least one
Results
From the original 135 meta-analyses retrieved, we could not establish whether grey literature was included in 12 cases (11 poorly referenced, one not retrieved). An additional eight meta-analyses were excluded because they did not fit the definition of a meta-analysis. Of the remaining 115, 38 (33%) were found to contain at least one item of grey literature. Five of these meta-analyses failed to meet the inclusion criteria. For all subsequent investigations, we used the sample of 33
Discussion
Our results suggest that the exclusion of grey literature from meta-analysis may result in an overestimate of an intervention effect by an average of 12%. In respect of quinine for nocturnal leg cramp, Hing and colleagues published a meta-analysis of four published studies.11 These investigators then repeated their meta-analysis with three previously unidentified US Food and Drug Administration (FDA) documents. They report that the published trials, compared with the FDA documents, consistently
References (16)
- et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
(1998) - et al.
Empirical evidence of bias in infertility research: overestimation of treatment effect in crossover trials using pregnancy as the outcome measure
Fertil Steril
(1996) - et al.
Publication bias in clinical research
Lancet
(1991) A dictionary of epidemiology
(1995)- et al.
Should unpublished data be included in meta-analyses?
JAMA
(1993) - et al.
Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals
JAMA
(1998) - et al.
Statistical power, sample size, and their reporting in randomized controlled trials
JAMA
(1994) - et al.
Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials
JAMA
(1995)
Cited by (618)
Primary vs grey: A critical evaluation of literature sources used to assess the impacts of offshore wind farms
2024, Environmental Science and PolicyCatheter-based renal artery denervation: facts and expectations
2023, European Journal of Internal MedicineWorking Together: How Academic Librarians Can Help Researchers Prepare for a Grey Literature Search for Systematic Reviews Involving Minoritized Populations
2023, Journal of Academic LibrarianshipEfficacy of imagery rescripting in treating mental disorders associated with aversive memories – An updated meta-analysis
2023, Journal of Anxiety DisordersTrial Registry Searches In Plastic Surgery Systematic Reviews: A Meta-epidemiological Study
2023, Journal of Surgical ResearchThe prevalence of cyber dating abuse among adolescents and emerging adults: A meta-analysis
2023, Computers in Human Behavior