Methods of Systematic Reviews and Meta-AnalysisCochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy
Introduction
The first systematic review relevant to physiotherapy was published in the early 1980s [1], [2], [3]. Since then, growth in the number of systematic reviews relevant to physiotherapy has been exponential [4]. Although we know that the quality of randomized controlled trials in physiotherapy is increasing with time [4], the quality of the systematic reviews relevant to physiotherapy has not been explored. Methods used in systematic reviews of physiotherapy interventions, including how the literature is searched, how the quality of trials is assessed, and how data from multiple trials are pooled, have not been systematically examined.
In other areas of healthcare, diverse review methods are used [5]. For example, quality assessment of studies occurred in only 31% of reviews of emergency medicine [5] and in 51% of reviews of pediatric complementary and alternative medicine [6]. The choice of review methods may have impact on the validity of the conclusions reached.
Previous examination of the quality of systematic reviews in the medical literature has identified important distinctions between Cochrane reviews and other systematic reviews. Jadad et al. found that Cochrane reviews were more likely to report criteria for including and excluding trials, assess the quality of trials, and be updated [7]. The quality of Cochrane and non-Cochrane reviews was also compared by Shea et al. [8] using two quality assessment tools: the Overview Quality Assessment Questionnaire (OQAQ) [9] and the Sacks checklist [10]. These analyses confirmed that Cochrane reviews were more likely to report inclusion/exclusion criteria for trials and were less likely to report a comprehensive search strategy, but were not different in overall quality [8]. In contrast, two more recent comparisons of Cochrane vs. non-Cochrane systematic reviews reported that Cochrane reviews fulfilled 2.5 more items, on average, on the Quality of Reporting of Meta-analyses (QUOROM) checklist [11], [12], and had a 3-point higher median score on the OQAQ [13].
Several scales and checklists have been used to assess the quality of systematic reviews—see the review by Shea et al. for a comparison of these [14]. Although some have face validity, OQAQ also has construct validity [9], [15]. However, the reliability of OQAQ has not been established.
This study aimed to describe the quality and the methods used in systematic reviews in physiotherapy. Specifically, the study examined which databases were searched, whether the grey literature was searched [16], how trial quality was assessed, how variables were reported, and the methods used to pool data and draw conclusions. In addition, this study aimed to compare the methods used in Cochrane and non-Cochrane reviews relevant to physiotherapy and to determine whether the Cochrane reviews relevant to physiotherapy were superior in quality to the non-Cochrane reviews. Secondary aims were to establish the interrater reliability of the OQAQ quality assessment tool for systematic reviews, and evaluate how quality has changed with time.
Section snippets
Methods
PEDro (Physiotherapy Evidence Database) (www.pedro.org.au) provides an index of randomized trials, systematic reviews, and evidence-based practice guidelines in physiotherapy. These are located using sensitive search procedures (http://www.pedro.org.au/criteria.html) applied to four major databases (MEDLINE [Medical Literature Analysis and Retrieval System Online], EMBASE [Excerpta Medica Database], CINAHL [Cumulative Index to Nursing and Allied Health Literature], and PsycINFO) and two
Results
There were 137 non-Cochrane reviews and 63 Cochrane reviews (including eight published outside the Cochrane Library). In terms of subdiscipline of physiotherapy, 62 were most relevant to musculoskeletal physiotherapy, 31 to cardiothoracics, 23 to neurology, 15 to continence and women's health, 13 to gerontology, 11 to orthopedics, 10 to pediatrics, four to ergonomics and occupational health, four to sports physiotherapy, and 27 to other areas of physiotherapy. The reviews were published between
Discussion
Most systematic reviews of physiotherapy interventions specify the databases searched to identify trials and assess the quality of included trials. Reviews conducted under the Cochrane Collaboration editorial system searched more databases and were more likely to have assessed trial quality, reported dichotomous outcomes for individual trials, and pooled data in a meta-analysis. Non-Cochrane reviews were more likely to conclude that there was a beneficial effect of treatment. Cochrane reviews
Conclusion
Cochrane systematic reviews appear to use more rigorous methods (including searching more databases and assessing trial quality) and are generally of a higher quality than non-Cochrane systematic reviews. The lower quality of non-Cochrane reviews may be associated with poorer control of bias. The quality of systematic reviews in physiotherapy is improving. Some avenues for further improving the methods used to perform systematic reviews have been identified.
Acknowledgments
This research was funded by a grant from the Motor Accidents Authority of New South Wales, Australia. The contribution of Natasha Lannin, Steve Kamper, and Julia Cameron, who assisted with data extraction, is gratefully acknowledged. Chris Maher and Rob Herbert's research fellowships are funded by the National Health and Medical Research Council of Australia.
References (43)
Does patient education in chronic disease have therapeutic value?
J Chronic Dis
(1982)- et al.
Evidence for physiotherapy practice: a survey of the Physiotherapy Evidence Database (PEDro)
Aust J Physiother
(2002) - et al.
Evaluating the quality of systematic reviews in the emergency medicine literature
Ann Emerg Med
(2001) - et al.
Validation of an index of the quality of review articles
J Clin Epidemiol
(1991) - et al.
Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses
Lancet
(1999) - et al.
The reporting quality of meta-analyses improves: a random sampling study
J Clin Epidemiol
(2008) - et al.
Assessing the quality of reports of randomized clinical trials: is blinding necessary?
Control Clin Trials
(1996) - et al.
Acupuncture and chronic pain: a criteria-based meta-analysis
J Clin Epidemiol
(1990) - et al.
A method for assessing the quality of a randomized control trial
Control Clin Trials
(1981) - et al.
The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus
J Clin Epidemiol
(1998)
Effectiveness of health promotion programs to increase motor vehicle occupant restraint use among young children
Am J Prev Med
An introduction to evidence-based practice for hand therapists
J Hand Ther
Conservative therapy for low back pain. Distinguishing useful from useless therapy
JAMA
The experimental evidence for weight-loss treatment of essential hypertension: a critical review
Am J Public Health
Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine
BMC Pediatr
Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals
JAMA
A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals
Eval Health Prof
Meta-analyses of randomized controlled trials
N Engl J Med
The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal
Crit Care Med
Assessing the quality of reports of meta-analyses: a systematic review of scales and checklists
Checklists for review articles
BMJ
Cited by (158)
Study found increasing use of core outcome sets in Cochrane systematic reviews and identified facilitators and barriers
2024, Journal of Clinical EpidemiologyCompared to what? An analysis of comparators in trials informing the National Institute of Clinical Excellence (NICE) low back pain guideline
2023, Brazilian Journal of Physical TherapyClinical trial registration has become more prevalent in physical therapy but it is still inadequate: A meta-research study
2023, Musculoskeletal Science and PracticePresentation approaches for enhancing interpretability of patient-reported outcomes in meta-analyses: a systematic survey of Cochrane reviews
2023, Journal of Clinical EpidemiologyEpidemiological, methodological, and statistical characteristics of network meta-analysis in anaesthesia: a systematic review
2023, British Journal of Anaesthesia