Background This study explains how to compare the relative bias of instrumental variable and conventional regression using negative controls and bias plots. Conventional observational analyses such as multivariable adjusted regression depend on the assumption of no unmeasured confounding. This assumption is rarely plausible, and results from observational studies have frequently been inconsistent with results from randomised controlled trials. Instrumental variable analyses can provide consistent estimates of causal effects in the presence of unmeasured confounding. Instrumental variables are defined by three assumptions, they must: 1) be associated with the exposure of interest, 2) share no common cause with the outcome, and 3) have no direct effects on the outcome except through the exposure of interest. However, regulators and clinicians find it difficult to interpret conflicting evidence from instrumental variable compared with conventional regression analyses and need to assess which approach is likely to be less biassed.

Methods In this paper we describe three techniques that can help answer this question: negative control outcomes, negative control populations and covariate balance tests. We illustrate these methods using an analysis of the effects of varenicline versus nicotine replacement products in primary care using data from 175,140 patients in the Clinical Practice Research Datalink. These patients were prescribed between 1 September 2006 and 31 October 2011.

Results Patients prescribed varenicline were more healthy in terms of almost all baseline characteristics. For example, they were younger (mean age differences in years = 1.66: 95% confidence interval (95% CI): 1.49, 1.84), visited the GP less often (mean difference in attendance per year = 5.82: 95% CI: 5.65, 5.99) and were less likely to have neuropsychiatric co-morbidities such as depression (risk difference per 100 patients treated = 2.57: 95% CI: 2.31, 2.83). The proposed instrumental variable, physicians’ previous prescription, was less associated with each of these baseline covariates (p_{heterogeneity} = 0.004, 5.84E-04, and 0.07 respectively). This suggests instrumental variable estimates of the effects of varenicline are likely to be less biassed than conventional methods.

Discussion Clinicians and regulators struggle to interpret conflicting evidence from instrumental variable compared with conventional regression analysis. The relative bias of these methods can and should be assessed using negative control populations and outcomes. The relative bias of instrumental variable and conventional analysis should be assessed using observed covariates. Researchers should report covariate balance plots with confidence intervals to robustly assess the relative bias for each covariate.