Publication bias is one the major drawbacks of meta-analysis. Its assessment is a recommendation stated in the detailed items by JAMA’s guides,¹ in the declaration QUOROM (quality of reporting of meta-analysis) for meta-analysis of experimental studies,² and MOOSE (meta-analysis of observational studies in epidemiology) for meta-analysis of observational studies.³

In practice few meta-analyses have assessed or adjusted for the presence of publication bias. A recent assessment of the quality of systematic reviews reported that only 6.5% and 3.2% of studies in both high impact general and specialist journals, respectively, had used a procedure to detect publication bias (Tallon et al, quoted in Sutton et al⁴). We have not found any previous report analysing the determinants of the assessment of publication bias. This is the major goal of this report, to examine variables related to publication bias ascertainment in a sample of systematic reviews with meta-analysis on cardiovascular diseases.

METHODS

A search for meta-analysis on cardiovascular diseases published in journals included in Medline was performed. Sample size was pre-estimated for a different goal from the one of this report, to detect determinants of publication bias: to find out a statistically significant (α = 0.05) relative risk of 2 between an hypothetical determinant of publication bias (with a frequency of 25%), assuming a statistical power of 80%, and a frequency of publication bias in the unexposed group of 20%, it was calculated that 232 meta-analyses would be required.

A preliminary search in Medline crossing the MeSH terms “cardiovascular diseases” and (“meta-analysis” or “systematic reviews”) yielded 730 meta-analyses from 1990 to 2002. Therefore, it was decided to collect all the systematic reviews with meta-analysis published in the even years (1990, 1992, … 2002). With the same criterion the Cochrane Library was also consulted. In both sources there were 436 references. Apart from the year of publication and the disease, additional inclusion criteria were: (a) to have the structure of a scientific systematic review with methods describing searching strategy, inclusion criteria, procedures for combining, etc (106 were excluded because they did not accomplish this structure; (b) that the meta-analysis should ascertain exposure-outcome associations (nine papers did not analyse any associations); (c) to have three or more primary studies with relevant information, excluding reports with zero effects in both index and reference groups (23 reviews had less than three studies); and (d) to provide the strength of the association and either its 95% confidence interval or the standard error of each primary study or the crude data to estimate them (73 papers did not provide these data). There were finally included 225 systematic reviews. No inclusion criteria were applied according to type of effect—that is, binary and continuous outcomes were included.

The unit of analysis has been “paper” and not “meta-analysis”. Frequently one systematic review contains more than one meta-analysis, although the methods (from which the data on assessment of publication bias are obtained) are the same. Systematic reviews were assessed by two non-blinded observers. Disagreement for every variable occurred in less than 10% of papers: in these situations and, after a reread of the conflicting item, an agreed answer was achieved. The outcome was the assessment of the publication bias by the original authors: it was considered as positive if in the review any method for detecting publication bias (visual inspection of funnel plots, regression analyses, trim-and-fill, fail-safe-N, etc) was described and its results reported or commented on. The assessment of publication bias has been related to several variables:

1. Characteristics of the publication: year of publication, country of the first author, type of journal (medicine in general, cardiology, epidemiology/public health, other medical specialties, and Cochrane Library), impact factor (according to the Journal of Citation Reports in the year before publication of the systematic review), epidemiologist and/or statistician among the authors (according to their affiliations), and the number of primary studies included in meta-analysis.

2. Aspects of the search and inclusion criteria: search in databases (Medline, Embase, other databases, several databases), handsearch in references of articles, search using other languages than English, search of unpublished reports, consultation with researchers to identify unpublished studies, search in scientific meetings, type of studies included in meta-analysis (of observational studies compared with experimental), and study quality in the inclusion criteria.

3. Characteristics of quality evaluation of primary studies by the original authors: was there quality ascertainment of primary studies? Was a published questionnaire applied in quality evaluation? If a published questionnaire was used, was it modified with the inclusion of ad hoc questions on the association? Was there more than one reviewer in quality evaluation?

4. Characteristics of the analysis: aspects of heterogeneity (was it assessed?, was there heterogeneity?, was it justified by stratification, meta-regression or any other method?, was it mentioned in the Discussion?), statistical significance of the pooled estimate, and type of statistical model (random or fixed effects). As some of the above mentioned variables require the extraction of figures of the pooled analysis and frequently more than one meta-analysis in a systematic review is reported, just one meta-analysis was selected with the next criteria, in decreasing order of importance: the analysis dealing with the main objective of the paper (giving precedence to composite outcomes over non-composite ones, harder—for example, mortality—outcomes over soft ones, and to clinical outcomes over surrogate outcomes/markers), the one with more studies, and the one with the most frequent outcome. If in a meta-analysis both RR and RD (risk difference) were given, RR was chosen; if both fixed effects and random effects models were offered, the former was collected. If a meta-analysis did not give information on heterogeneity, it was estimated under the fixed effects model.

Crude associations have been ascertained by the relative risk (RR) and its 95% confidence interval (CI). To identify the independent predictors of the assessment of publication bias we have applied multivariate stepwise logistic regression analysis, using a forward model with probabilities of entry and removal of 0.2 and 0.21, respectively, with the objective of keeping variables with p<0.2 into the model. This cut off has been suggested as appropriate for retaining in a multivariate model relevant confounders.⁵ Multivariate logistic regression analysis yields adjusted odds ratios (OR). The stepwise model obtained with all the sample was applied to reviews with eight or more primary studies, as an intermediate cut off found in other reviews assessing publication bias in series of meta-analyses.^4,6–8

Statistical analyses have been carried out with Stata 8-SE (Stata Corporation, College Station, TX).

RESULTS

Table 1 shows the description of the 225 of the sample of reviews with meta-analysis included in this report. Most of them come from USA and Canada (85, 35.8%), in cardiology journals (66, 29.3%), and are published in journals with an impact lower than that of the British Medical Journal (171, 76.0%). There is an epidemiologist among the authors in 83 (36.9%) reviews, and a statistician in 30 (13.2%). Publication bias is assessed in 25 (11.1%) reviews by the next procedures: funnel plot (n = 11), Begg-Mazumdar (n = 5), Egger (n = 5), funnel plot regression (n = 1), and trim and fill (n = 3). No review gives results adjusting for the existence of publication bias.

Table 2 shows the association of assessment of publication bias with variables related to the publication. The ascertainment of publication bias increases with time of publication: from 3.4% in reviews published before or during 1998 to 19.0% in 2002. There is no relation with the first author’s country, impact factor of the journal, and the existence of epidemiologist or statistician among the authors. In systematic reviews published in the Cochrane Library there is a borderline statistically significant lower frequency of assessment of publication bias. An increase in the assessment of publication bias with the number of primary studies is observed and in systematic reviews of observational studies compared with experimental studies (RR = 2.98, 95% CI = 1.44 to 6.16).

Table 3 shows the association of assessment of publication bias with characteristics of the search and inclusion criteria of primary studies. There is no significant relation with a search into the most popular databases (Medline, Embase), however, an increased frequency of assessment is found when authors search other databases (RR = 2.50, 95% CI = 1.09 to 5.76). This latter fact is also reflected with the number of databases consulted: more than two databases augments the assessment of bias significantly. Similar relations are found by hand search in references of articles, albeit with borderline significance. There is no relation with search in other languages than English and non-published reports. We found no association with the quality evaluation of primary studies and these results are not shown.

Table 4 shows the association of assessment of publication bias with variables related to the analysis. The frequency increases with heterogeneity ascertainment (RR = 4.88, 95% CI = 0.68 to 35.0), and with the presence of heterogeneity (RR = 3.03, 95% CI = 1.18 to 7.79). The existence of statistical significance in the pooled estimate improves, but not significantly, bias assessment. The same occurs when the statistical model (either random or fixed effects) is mentioned.

Table 5 shows the results of the stepwise logistic regression model. The significant variables related to an increased assessment of publication bias are: number of primary studies in meta-analysis, number of searched databases, date of publication, and to be a review of observational studies. Some types of journal (other medical specialties apart from cardiology) and Cochrane Library are associated with a lower assessment of publication bias. The presence of a statistician among the authors and to mention the statistical model augment the frequency of bias assessment, albeit with borderline statistical significance. This model was applied to the subgroup of systematic reviews with eight or more studies. The results are roughly the same, apart from a clear decrease in the association with the number of primary studies, and loss of statistical significance in several variables because of a smaller sample size.

DISCUSSION

First of all we comment on the limitations of this study. Our inclusion criteria may introduce a selection bias if meta-analyses published in the odd years (we selected publication in the even years) differ in their methods; we believe it is unlikely (the number of meta-analysis in the odd years in Medline was a little less than in the even years, 353 compared with 367). Meta-analyses having less than three studies were also discarded because there are no statistical procedures to detect publication bias when the number of primary studies is low. An important number of meta-analysis was discarded as they did not provide enough information of the primary studies to repeat the analysis. This was done to study the influence of the pooled statistical results (statistical significance of both the pooled estimate and heterogeneity tests) in the assessment of publication bias; otherwise these variables could not have been analysed. We believe that this inclusion criterion does not introduce a significant bias as other methodological characteristics (quality of the primary studies, etc) are unrelated to the assessment of bias in our results. Nevertheless, our inclusion criteria excluded reviews that did not pool results; it may imply the included reviews are, in general, of better quality. Reviews of higher quality are usually methodologically sound studies, which may relate to the ability to pool results, and the assessment of publication bias might be higher in our series than in a representative sample of all systematic reviews.

The assessment questionnaire applied to systematic reviews mostly consists of objective questions. Nevertheless, some items can be considered rather subjective, such as the identification of epidemiologist and statistician among the authors. This was done according to the authors’ affiliations; it may happen that authors with enough knowledge of epidemiology/biostatistics affiliated to clinical departments is unnoticed. This can produce a non-differential misclassification bias, yielding a closer to null relative risk, although a non-differential one cannot be discarded if the non-identification of these authors would be dependent upon some other variable, such as the journal of publication, country of origin, etc. This fact may help to justify why in crude analysis neither the presence among the authors of an epidemiologist nor a statistician do not increase significantly the frequency of bias assessment, whereas in the multivariate model a statistician among the authors augments bias assessment with borderline significance. It has been reported that an epidemiologist or a statistician improves the quality of statistical methods in clinical trials.⁹

Another limitation for the identification of factors related to the assessment of publication bias is the low frequency of bias assessment, which implies lack of statistical power for some associations. With more reviews it may happen that some borderline associations would reach statistical significance.

Our study is focused on cardiovascular diseases. Cardiovascular diseases are the principal cause of mortality and the research methods on them do not differ from those applied to study other diseases. Therefore we think that these results can be generalised to other diseases.

It is true that most of the tests applied for detecting publication bias can have another interpretation as these methods, when the results are positive, show different effects between smaller and larger studies: what we have ascertained has been the use of these procedures with the goal of detecting publication bias. It is interesting to point out that adjustments in the analysis for publication bias were never performed in our sample of systematic reviews.

Our results cannot be compared with other studies, as we have failed to find any similar studies. Since the dissemination of meta-analytical methods, the importance of identification of publication bias has been emphasised.^10,11 Nevertheless, the assessment of publication bias is scarce in our sample. It has improved with time, and this may have been influenced by consensus statements, such as QUOROM² and MOOSE.³ Anyway, the frequency of publication bias assessment is currently low.

A surprising result is the low frequency of publication bias assessment in reviews published in the Cochrane Library. Cochrane reviews usually follow a rigorous protocol of information search that should lead to a lower frequency of publication bias. Besides, the frequency of significant results in the pooled estimate of Cochrane reviews is lower than in meta-analyses published in conventional journals (29.7% compared with 75.8%, p<0.001). As publication bias is related to significant results, this may be another reason to explain the low assessment of publication bias in Cochrane reviews. However, the inclusion of the statistical significance of the pooled estimate in the multivariate logistic regression model does not change the results.

Publication bias is more frequently ascertained in reviews of observational studies than in reviews of experimental studies. This may be influenced by the work of Easterbrook et al,¹² in which it is reported that observational studies are more prone than experimental studies to publication bias due to statistical significance of their results.

The number of primary studies in meta-analysis is clearly associated with a higher frequency of bias assessment. This result is sensible as when there are few studies it is difficult to statistically ascertain the presence of bias. However, another fact can also influence this result: the statistical power of each primary study. In fact, statistical power can be related to the number of studies: it may happen that if there are few studies, although with appropriate power, there is no need to carry out more research.

The search characteristics are related to bias assessment. A higher frequency of bias assessment is found with a higher number of consulted databases. This may reflect a more thorough search and a stronger concern with publication bias by the authors. However, it is interesting to point out that other facts also indicating a thorough search and more concern on publication bias (such as search in other languages than English and the search of unpublished reports) do not increase the assessment of publication bias.

Regarding the factors related to statistical analysis of meta-analysis, one variable is included in the multivariate logistic model, specification of the applied statistical model, with borderline significance. It reflects quality of meta-analysis, as it is included in most checklists.^1–3 A statistically significant pooled estimate increases, albeit not significantly, the assessment of bias in crude analysis, but it is not included in the multivariate model. This result is in a certain way unexpected, as statistical significance is one of the most important determinants of publication bias^12–16 and well known by most authors: it should launch the search of publication bias. Its exclusion in the multivariate model may reflect our lack of statistical power to detect moderate associations.

In crude analysis, the presence of heterogeneity increases the frequency of publication bias assessment. However, in multivariate analysis, this variable lost its significance despite a high OR. We cannot conclude anything, although it may suggest that there is an association between heterogeneity and assessment of publication bias. Caution is needed regarding this interpretation as heterogeneity may reflect that biased studies have included in the pooled analysis when they should have been discarded.^17,18 Furthermore, funnel plot and derived procedures (such as the trim and fill) are inappropriate to detect publication bias when heterogeneity exists.¹⁹

It is debatable whether publication bias should be assessed in all meta-analysis. In certain situations the search of publication bias may be not relevant from a practical point of view: when all studies report a significant, large effect, it does not really matter if small studies are not published (they will be overridden), or when large studies, with independence of their results, drive the meta-analysis. In these situations, although the consequences of publication bias are small, it may be still relevant to identify that publication bias exists. To know the factors associated with publication bias can help to improve the publication pattern of the future.

In conclusion, the main fact found in this report is the low frequency of assessment of publication bias in systematic reviews, what means that published guidelines are not completely adhered, although it has improved with time. The assessment of publication bias is related to the number of pooled studies, observational studies, and other characteristics of quality of a systematic review, either of the search of primary studies (number of searched databases), or the statistical analysis. A surprising result, which needs further confirmation, is the lower assessment in Cochrane reviews.