Methods of Systematic Reviews and Meta-Analysis
Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy

https://doi.org/10.1016/j.jclinepi.2008.09.018Get rights and content

Abstract

Objective

To describe the quality and methods of systematic reviews of physiotherapy interventions, compare Cochrane and non-Cochrane reviews, and establish the interrater reliability of the Overview Quality Assessment Questionnaire (OQAQ) quality assessment tool.

Study Design and Setting

A survey of 200 published systematic reviews was done. Two independent raters assessed the search strategy, assessment of trial quality, outcomes, pooling, conclusions, and overall quality (OQAQ). The study was carried out in the University research center.

Results

In these reviews, the five most common databases searched were MEDLINE, EMBASE, Cochrane Library, CINAHL, and Cochrane Review Group Registers. The Cochrane allocation concealment system and Jadad Scale were most frequently used to assess trial quality. Cochrane reviews searched more databases and were more likely to assess trial quality, report dichotomous outcomes for individual trials, and conduct a meta-analysis than non-Cochrane reviews. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews were of higher quality than non-Cochrane reviews. There has been an increase in the quality of systematic reviews over time. The OQAQ has fair to good interrater reliability.

Conclusion

The quality of systematic reviews in physiotherapy is improving, and the use of Cochrane Collaboration procedures appears to improve the methods and quality.

Introduction

The first systematic review relevant to physiotherapy was published in the early 1980s [1], [2], [3]. Since then, growth in the number of systematic reviews relevant to physiotherapy has been exponential [4]. Although we know that the quality of randomized controlled trials in physiotherapy is increasing with time [4], the quality of the systematic reviews relevant to physiotherapy has not been explored. Methods used in systematic reviews of physiotherapy interventions, including how the literature is searched, how the quality of trials is assessed, and how data from multiple trials are pooled, have not been systematically examined.

In other areas of healthcare, diverse review methods are used [5]. For example, quality assessment of studies occurred in only 31% of reviews of emergency medicine [5] and in 51% of reviews of pediatric complementary and alternative medicine [6]. The choice of review methods may have impact on the validity of the conclusions reached.

Previous examination of the quality of systematic reviews in the medical literature has identified important distinctions between Cochrane reviews and other systematic reviews. Jadad et al. found that Cochrane reviews were more likely to report criteria for including and excluding trials, assess the quality of trials, and be updated [7]. The quality of Cochrane and non-Cochrane reviews was also compared by Shea et al. [8] using two quality assessment tools: the Overview Quality Assessment Questionnaire (OQAQ) [9] and the Sacks checklist [10]. These analyses confirmed that Cochrane reviews were more likely to report inclusion/exclusion criteria for trials and were less likely to report a comprehensive search strategy, but were not different in overall quality [8]. In contrast, two more recent comparisons of Cochrane vs. non-Cochrane systematic reviews reported that Cochrane reviews fulfilled 2.5 more items, on average, on the Quality of Reporting of Meta-analyses (QUOROM) checklist [11], [12], and had a 3-point higher median score on the OQAQ [13].

Several scales and checklists have been used to assess the quality of systematic reviews—see the review by Shea et al. for a comparison of these [14]. Although some have face validity, OQAQ also has construct validity [9], [15]. However, the reliability of OQAQ has not been established.

This study aimed to describe the quality and the methods used in systematic reviews in physiotherapy. Specifically, the study examined which databases were searched, whether the grey literature was searched [16], how trial quality was assessed, how variables were reported, and the methods used to pool data and draw conclusions. In addition, this study aimed to compare the methods used in Cochrane and non-Cochrane reviews relevant to physiotherapy and to determine whether the Cochrane reviews relevant to physiotherapy were superior in quality to the non-Cochrane reviews. Secondary aims were to establish the interrater reliability of the OQAQ quality assessment tool for systematic reviews, and evaluate how quality has changed with time.

Section snippets

Methods

PEDro (Physiotherapy Evidence Database) (www.pedro.org.au) provides an index of randomized trials, systematic reviews, and evidence-based practice guidelines in physiotherapy. These are located using sensitive search procedures (http://www.pedro.org.au/criteria.html) applied to four major databases (MEDLINE [Medical Literature Analysis and Retrieval System Online], EMBASE [Excerpta Medica Database], CINAHL [Cumulative Index to Nursing and Allied Health Literature], and PsycINFO) and two

Results

There were 137 non-Cochrane reviews and 63 Cochrane reviews (including eight published outside the Cochrane Library). In terms of subdiscipline of physiotherapy, 62 were most relevant to musculoskeletal physiotherapy, 31 to cardiothoracics, 23 to neurology, 15 to continence and women's health, 13 to gerontology, 11 to orthopedics, 10 to pediatrics, four to ergonomics and occupational health, four to sports physiotherapy, and 27 to other areas of physiotherapy. The reviews were published between

Discussion

Most systematic reviews of physiotherapy interventions specify the databases searched to identify trials and assess the quality of included trials. Reviews conducted under the Cochrane Collaboration editorial system searched more databases and were more likely to have assessed trial quality, reported dichotomous outcomes for individual trials, and pooled data in a meta-analysis. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews

Conclusion

Cochrane systematic reviews appear to use more rigorous methods (including searching more databases and assessing trial quality) and are generally of a higher quality than non-Cochrane systematic reviews. The lower quality of non-Cochrane reviews may be associated with poorer control of bias. The quality of systematic reviews in physiotherapy is improving. Some avenues for further improving the methods used to perform systematic reviews have been identified.

Acknowledgments

This research was funded by a grant from the Motor Accidents Authority of New South Wales, Australia. The contribution of Natasha Lannin, Steve Kamper, and Julia Cameron, who assisted with data extraction, is gratefully acknowledged. Chris Maher and Rob Herbert's research fellowships are funded by the National Health and Medical Research Council of Australia.

References (43)

  • D.C. Grossman et al.

    Effectiveness of health promotion programs to increase motor vehicle occupant restraint use among young children

    Am J Prev Med

    (1999)
  • J.C. MacDermid

    An introduction to evidence-based practice for hand therapists

    J Hand Ther

    (2004)
  • R.A. Deyo

    Conservative therapy for low back pain. Distinguishing useful from useless therapy

    JAMA

    (1983)
  • M.F. Hovell

    The experimental evidence for weight-loss treatment of essential hypertension: a critical review

    Am J Public Health

    (1982)
  • D. Moher et al.

    Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine

    BMC Pediatr

    (2002)
  • A.R. Jadad et al.

    Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals

    JAMA

    (1998)
  • B. Shea et al.

    A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals

    Eval Health Prof

    (2002)
  • H.S. Sacks et al.

    Meta-analyses of randomized controlled trials

    N Engl J Med

    (1987)
  • A. Delaney et al.

    The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal

    Crit Care Med

    (2007)
  • B. Shea et al.

    Assessing the quality of reports of meta-analyses: a systematic review of scales and checklists

  • A.D. Oxman

    Checklists for review articles

    BMJ

    (1994)
  • Cited by (158)

    View all citing articles on Scopus
    View full text