Article Text

Download PDFPDF

P41 Assessment of publication bias in systematic reviews of health services and delivery research
  1. AA Ayorinde1,
  2. I Williams2,
  3. R Mannion2,
  4. F Song3,
  5. M Skrybant4,
  6. RJ Lilford1,
  7. Y-F Chen1
  1. 1Warwick Centre for Applied Health Research and Delivery, University of Warwick, Coventry, UK
  2. 2Health Services Management Centre, University of Birmingham, Birmingham, UK
  3. 3Department of Population Health and Primary Care, University of East Anglia, Norwich, UK
  4. 4Institute of Applied Health Research, University of Birmingham, Birmingham, UK


Background Publication bias is a major threat to the validity of systematic reviews (SRs). Strategies to identify and reduce publication bias are routinely incorporated into SRs of clinical interventions, but the level of adoption of these strategies in SRs relating to health services and delivery research (HSDR) is unclear. The objectives of this study were to describe the characteristics of SRs of HSDR with regards to assessment of publication bias, and to evaluate factors associated with this.

Methods A stratified random sample of 200 SRs of quantitative HSDR published in English from 2007–2017 was selected from the Health Systems Evidence database. Half (100) of the selected reviews (intervention reviews) concern interventions to improve the effectiveness/efficiency of service delivery, which would mostly involve comparative studies. The other half (association reviews) evaluates associations between different variables along the service delivery causal chain, which often include observational studies. Data extracted included: any reference to publication bias, methods for detecting/mitigating publication bias or reasons for no assessment, number of included studies, inclusion of meta-analyses and whether the use of SR guideline was reported. Journals were divided into those that did or did not formally endorse specific SR guidelines such as PRISMA, and journal impact factors were obtained. Factors associated with assessment of publication bias were explored using multivariable logistic regression.

Results Of all 200 SRs, 48% commented on publication bias. However, only 25% formally assessed publication bias either through statistical analysis (mostly funnel plots) or as part of the quality assessment of included studies (e.g. the Cochrane risk of bias tools). Insufficient number of studies, heterogeneity and lack of pre-registered protocols were common impediments in assessing publication bias. In the multivariable analysis, assessment of publication bias was associated with SRs’ inclusion of a meta-analysis [odds ratio (OR) 9.08 (95% CI 3.67, 22.43)], reviewers reporting the use of SR guidelines [OR 4.09 (1.71, 9.78)], being an intervention review [OR 3.75 (1.42, 9.91)] and higher journal impact factor [OR 1.16 (1.02, 1.30)] but not significantly associated with number of studies included in SRs or journal endorsement of SR guidelines.

Conclusion Overall, the awareness of publication bias in HSDR reviews is comparable to that of reviews of clinical interventions, however, formal assessment of publication bias is less common especially in association reviews. This reflects the heterogeneity of HSDR evidence and the limits of current tools for assessing publication bias. Adherence to existing SR guidelines should be promoted.

  • Publication bias
  • systematic review
  • health services reseach

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.