How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study

Background Systematic reviews (SRs) are expected to critically appraise included studies and privilege those at lowest risk of bias (RoB) in the synthesis. This study examines if and how critical appraisals inform the synthesis and interpretation of evidence in SRs. Methods All SRs published in March–May 2012 in 14 high-ranked medical journals and a sample from the Cochrane library were systematically assessed by two reviewers to determine if and how: critical appraisal was conducted; RoB was summarised at study, domain and review levels; and RoB appraisals informed the synthesis process. Results Of the 59 SRs studied, all except six (90%) conducted a critical appraisal of the included studies, with most using or adapting existing tools. Almost half of the SRs reported critical appraisal in a manner that did not allow readers to determine which studies included in a review were most robust. RoB assessments were not incorporated into synthesis in one-third (20) of the SRs, with their consideration more likely when reviews focused on randomised controlled trials. Common methods for incorporating critical appraisals into the synthesis process were sensitivity analysis, narrative discussion and exclusion of studies at high RoB. Nearly half of the reviews which investigated multiple outcomes and carried out study-level RoB summaries did not consider the potential for RoB to vary across outcomes. Conclusions The conclusions of the SRs, published in major journals, are frequently uninformed by the critical appraisal process, even when conducted. This may be particularly problematic for SRs of public health topics that often draw on diverse study designs.


Observational Epidemiology
To synthesise current evidence on the prevalence of poor diet, inadequate physical activity, and overweight and obesity in prisoners.
Observational Observational Epidemiology To establish whether a difference in SBP between arms is associated with ipsilateral angio graphically proven subclavian stenosis on the side of the arm with the lowest pressure, with peripheral or cardiovascular disease, and with an increased risk of cardiovascular-related or all-cause mortality.
Cohort or crosssectional studies 28 Multiple Yes Lancet Hughes [22] Observational Epidemiology To identify the characteristics and coverage of research for the prevalence and risk of violence against adults with disabilities; assess the quality of this research; and synthesise evidence on the prevalence and risk of violence against adults with disabilities to identify knowledge gaps and research priorities. Observational Epidemiology To evaluate the effects of using alternative statistical presentations of the same risks and risk reductions on understanding, perception, persuasiveness and behaviour of health professionals, policy makers, and consumers.
Randomized and nonrandomized controlled parallel and cross-over studies. 35 Multiple Yes RCTs = Randomised-controlled trials; CRCTs = Cluster randomised-controlled trials; CCTs = Controlled clinical trial Appendix  No. Table 2 summarises ind criteria ax for each studytext says two of the studies are of low methodological quality but no reasons provided. All 6 included studies demonstrated adequate allocation, concealment, blinding, percentage participation, and comparability of groups both at baseline and in provision of care apart from the intervention; however, 3 studies did not report the method of randomization (Table 2). We therefore performed a sensitivity analysis excluding these studies.

Pediatrics Mohamed [40] Yes
Weighted score (with domains allocated one or two points) None Score based on summing except two domains marked out of 2 (case and control comparability, ascertainment of exposure) We planned to perform sensitivity analysis on the primary outcomes based on trial quality, separating highquality trials from trials of lower quality. For the purposes of this sensitivity analysis, we defined high quality as a trials rated as 'low risk of bias' for sequence generation and allocation concealment.

Fedorowicz [57] Yes
Low risk of bias requires all criteria to be met None Not by prioritising domain. "The overall risk of bias of each of the included studies has also been reported according to the following categories.
• Low risk of bias (plausible bias unlikely to seriously alter the results) if all criteria were met.
• Unclear risk of bias (plausible bias that raises some doubt about the results) if one or more criteria were assessed as unclear.
• High risk of bias (plausible bias that seriously weakens confidence in the results) if one or more criteria were not met.

Cochrane
Han [58] Yes Unclear None Three of the four included studies were at moderate to high risk of bias and one study (Grant 2011) was at low to moderate risk of bias. Magnitude and direction of bias rationale not stated.

Cochrane
Akl [59] Yes At least two of the four criteria required None "We also conducted pre-planned sensitivity analyses excluding studies: • of lower methodological quality (ie those which did not meet at least two of the four methodological criteria);"