Article Text

Download PDFPDF

P39 Intervention differential effects and threshold selection: an evaluation of methods illustrated in weight-management studies
  1. LS Beggs1,2,3,
  2. R Briscoe1,2,4,
  3. C Griffiths1,5,
  4. GTH Ellison1,2,
  5. MS Gilthorpe1,2
  1. 1Leeds Institute for Data Analytics, University of Leeds, Leeds, UK
  2. 2School of Medicine, University of Leeds, Leeds, UK
  3. 3Centre for Health Technology Evaluation, National Institute for Health and Care Excellence, Manchester, UK
  4. 4Leeds Teaching Hospitals Trust, Leeds Teaching Hospitals Trust, Leeds, UK
  5. 5Sports Science, Leeds Beckett University, Leeds, UK


Background Intervention differential effects (IDE) occur when change in a health outcome following an intervention depends upon the baseline value of that outcome. Oldham’s method and multilevel modelling are methods used to detect IDEs. However, the conditions under which these methods are robust are not well documented. One condition that has not been explored is detection of IDEs in studies which recruit according to baseline health status (i.e. above a threshold). We hypothesised that recruiting/selecting above a threshold affects the reliability of existing methods to detect IDEs because of regression to the mean. We hypothesised that comparing these ‘truncated’ samples with a control group restores the robustness of these methods. Using weight-loss interventions as an example, we show how to overcome the challenges of regression to the mean in studies with threshold selection criteria.

Methods We simulated two datasets comprising repeated measures of body mass index (BMI) data for 1000 males aged 25–34 (‘population’ datasets). One dataset was simulated to have an IDE, and the other (‘null’) dataset was simulated without. Half the population in each dataset were simulated to receive a weight-loss intervention. To emulate real-word weight-loss interventions, we truncated each population dataset to select intervention and control group samples with BMI scores above ≥30 kg/m2. Oldham’s method and multilevel modelling were used on the ‘population’ intervention groups and corresponding ‘truncated’ samples for each simulation. We repeated each analysis to contrast the intervention and control group datasets (using Fisher’s z-transformation and student’s t-test for Oldham’s method, and the likelihood ratio test for multilevel modelling). Simulations were repeated 10 000 times to generate Type I error rates and 95% credible intervals. Simulations were performed in R and MLwiN.

Results Under the null of no IDE, Oldham’s method and the multilevel model yielded Type I error rates >90%, confirming that selecting above a threshold leads to bias due to regression to the mean. Type I error rates returned to 5% for the multilevel model when a control group was introduced and the likelihood ratio test employed, while Type I error rates improved but remained elevated when Fisher’s z-transformation and student’s t-test were used to contrast groups.

Conclusion Our study shows that multilevel models can robustly detect IDEs in ‘truncated’ samples (selected above a threshold) if analyses involve a control group. For study designs that do not collect control group data (such as most evaluations of weight management programmes), the identification of IDEs currently remains intractable.

  • Methods
  • body weight
  • intervention studies

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.