Background Systematic reviews of evidence frequently help inform research, clinical guidelines, and policy. Although randomised controlled trials (RCTs) are ideal, many areas of public health rely on evidence from non-randomised studies which are more susceptible to bias. ‘Risk Of Bias In Non-Randomised Studies of Interventions’ (ROBINS-I) is a widely used critical appraisal tool developed by Cochrane in 2016, which assesses risk of bias on an absolute scale, such that a low risk-of-bias study is equivalent to a well-conducted RCT. ROBINS-I has been seen as a major methodological innovation, but its complexity has led to concerns that it may be misapplied. We review for the first time how ROBINS-I is used in a sample of recent systematic reviews.

Methods Systematic reviews using ROBINS-I were identified by forward citation and keyword/abstract searches in six databases, restricted to January and February 2020. The review protocol was preregistered in PROSPERO (CRD42020170785). Reported ROBINS-I ratings and data on how ROBINS-I was used were extracted from each review. Methodological quality of reviews was assessed using AMSTAR 2 (‘A MeaSurement Tool to Assess systematic Reviews’). Mixed-effects partial proportional odds regression was used to assess associations between review characteristics (e.g. methodological quality and industry funding) and risk-of-bias ratings. Screening and quality appraisals were conducted independently by two reviewers.

Results Of 181 hits, 124 reviews were analysed with data extracted on 1,344 included studies. Risk of bias was reported as serious/critical for 54.8% of included studies, most commonly due to confounding, but 8.0% reported low risk of bias. Poorly conducted reviews were more likely to report lower risk-of-bias ratings, with an apparent dose-response relationship. Compared to reviews with moderate/high AMSTAR 2 rating, odds of low risk-of-bias ratings were higher in low-quality reviews (odds ratio: 1.89 [95% confidence interval: 0.36–9.94]), and considerably higher in critically low-quality reviews (4.70 [1.01–21.78]). Competing interests and industry funding were not uniformly predictive of higher or lower ratings, although these analyses had low statistical power. Deviations from the guidance of the tool were seen in 40.3% of studies, with 20.2% reporting ratings using a non-standard scale.

Discussion Systematic reviews conducted using Cochrane’s recommended tool for non-randomised studies may misleadingly suggest a robust evidence base exists when used by reviewers without adequate epidemiological expertise. This may lead to misleading conclusions, especially for public health guidelines. Greater training and expertise are required to ensure that widespread use of the tool does not lead to an increase in misleading reviews.