Background Frustrated with the onslaught of articles reporting fascination with results that appear paradoxical but are merely due to selection bias, we studied the apparent effect of obesity on diabetes risk in youth who had a test for diabetes. We hypothesised that obese subjects would have lower rates of diabetes than non-obese subjects due to selection bias, and consequently, obesity would appear to lower the risk of diabetes.
Methods Retrospective cohort study of children (4–9 years), pre-teens (10–12 years) and teenagers (13–19 years). Participation was restricted to those who had a test of haemoglobin A1C along with measured height and weight. Body mass index percentile via the Centers for Disease Control and Prevention age and sex standards was calculated and categorised. The main outcome was A1C%, subsequently categorised at the level for diagnosis of diabetes mellitus (≥6.5%).
Results The sample consisted of 134 (2%) underweight, 1718 (30%) healthy weight, 660 (12%) overweight and 3190 (56%) obese individuals. 16% (n=936) had an A1C≥6.5%. Overall, healthy weight children had 8.2 times the risk of A1C≥6.5% (95% CI 5.3 to 12.7) compared with those in the obese category. The relative risk was 13 in pre-teens (95% CI 8.5 to 20.0) and 3.9 in teenagers (95% CI 3.3 to 4.7).
Conclusions Healthy weight was associated with a 4–13 times higher relative risk of diabetes mellitus compared with being obese. While apparently shocking, the study’s fatal flaw (selection bias) explains the ‘paradoxical’ finding. Ignoring selection bias can delay advances in medical science.
- study design
- epidemiological methods
- research design in epidemiology
Statistics from Altmetric.com
Peer-reviewed medical journals are regularly publishing studies where the recruitment of subjects is based on an effect of the exposure of interest, a methodological error that may induce a form of selection bias.1–3 Selection bias can occur at the study design or analysis stage as a result of conditioning on a common effect of two variables, one of which is the exposure, or a variable associated with the exposure, and the other of which is the outcome, or a variable associated with the outcome.4 Selection bias may reverse the expected direction of an effect, producing results that appear dramatically contrary to expectations. These surprising results are not due to errors in previous research, random variation or statistical mistakes. Rather, the results are entirely predictable as they are due to a systematic error that operates occultly behind the scenes of the conduct of the rest of the study. If participants are chosen based on an effect of the study’s exposure variable, then that exposure variable can appear protective. The bias in the structure of these studies has been described in detail4–6 and can create associations that are in the opposite direction from that which occurs in the general population.
Since the results of studies with selection bias appear contrary to expectations, some call the results paradoxical. The most common of these is the so-called ‘obesity paradox’ which occurs when studies evaluate the effect of obesity among individuals who have a disease caused by obesity, such as heart failure, stroke or myocardial infarction. These studies report that among individuals with the disease the subjects who were obese fare better than those who were not obese.7–10 While a health risk such as obesity may confer some protective benefits, for example, stronger bones, the apparently protective benefits of obesity are due to selection bias when subjects are chosen based on an effect of obesity. Despite the known methodological explanation for this paradox,3 11 12 articles reporting evidence of an obesity paradox are on the rise. Our search of EMBASE for the phrase ‘obesity paradox’ in the title, abstract or keywords found that the first article appeared in the year 200213 and there are now >800 full-text peer-reviewed articles.14 While some of these studies describe the selection bias that causes the apparently paradoxical results, the vast majority express a fascination with the results and call for more research into the possible ways that obesity protects people from the hazards of diseases caused by obesity, such as heart failure.7–9
Frustrated with the onslaught of articles expressing fascination with these paradoxical results while not explaining to the reader that the results were, in fact, predictable and due to selection bias, we decided to conduct our own study. We chose to study diabetes mellitus in youth as this is an illness with two well-described mechanisms; type 1, due to lack of insulin production, and type 2, due to lack of insulin sensitivity which is a consequence of obesity.15 While diabetes in young people was historically thought to be only the result of type 1 diabetes, the paediatric obesity epidemic has led to an increasing number of cases of type 2 diabetes.16 17
Diabetes risk is often assessed by a person’s level of haemoglobin A1C (A1C), a measure of blood sugar control. As such, having an A1C test is an outcome of either being obese or being ill with symptoms and signs of diabetes. An A1C level ≥6.5% is generally considered diagnostic of diabetes.15 The purpose of our study was to evaluate the association of A1C level with weight status in children and adolescents ages 4–19 years. We hypothesised that among those with at least one A1C level obese subjects would have lower rates of diabetes than non-obese subjects due to selection bias, and consequently, obesity would appear to lower the risk of diabetes.
This project was a retrospective cohort study involving a chart-review analysis restricted to youth who had an A1C blood test drawn. Clinical data from medical records in the University of Minnesota Clinical Data Repository were compiled by data analysts through the Best Practices Integrative Informatics Consulting and Collaborative Science Core. Participants in our sample were patients who had (1) at least one A1C measure when they were between 4 and 19 years of age between 6 January 2011 and 4 August 2017, and (2) a height and weight measurement associated with this A1C. If there were multiple A1C measurements for one participant, only the first A1C measurement was included in the analysis.
The primary exposure in this study is age and sex-specific body mass index (BMI) percentile. BMI percentile was calculated for all participants from measured height and weight using the Centers for Disease Control and Prevention’s (CDC’s) SAS Program for the 2000 CDC Growth Charts.18 BMI percentile was broken into categories based on the CDC guidelines for defining childhood weight status where underweight is less than the 5th percentile, healthy weight is from the 5th to less than the 85th percentiles, overweight is from the 85th to less than the 95th percentiles, and obese is at the 95th percentile or greater.
The primary outcome in this study is haemoglobin A1C (%) as collected from the participants’ medical records. A1C levels were categorised as either <6.5% or ≥6.5%, the level for diagnosis of diabetes mellitus. Among those with diabetes, we also categorised those with a level ≥8% as having poorly controlled diabetes. Other demographics and sample characteristics examined in this sample included age, race, ethnicity, height, weight and blood pressure.
SAS Software V.9.4 (SAS Institute Inc) was used for analysis. Data were examined for outliers and implausible values using histograms and quantile plots. The CDC SAS program for calculating BMI percentile flagged n=24 participants with extreme or biologically implausible values for height, weight and BMI (using the following modified z-score cut-offs: BMI for age <−5 or >13, weight for age <−5 or >13, height for age <−5 or >5). These participants were excluded from the main analysis. The A1C values of the excluded participants were compared with those included via a sensitivity analysis using a two-sample t-test (excluded mean±SD: 5.35±0.72; included mean±SD: 6.04±1.76; P=0.06).
Descriptive statistics were calculated for participant demographics and health characteristics. Pearson’s correlation coefficients and simple linear regression were used to test for a linear association between continuous BMI percentile and haemoglobin A1C. Generalised linear regression models were used to compare relative and absolute contrasts of risk of a high A1C (≥6.5% as well as ≥8%) between underweight, healthy weight, overweight and obese participants. We assumed that age would modify the association of weight status and A1C and divided the cohort into children (4–9 years), pre-teens (10–12 years) and teenagers (13–19 years). Within these groups, there was no significant interaction by age. Results are reported as relative and absolute contrasts, and their 95% CIs for all ages together, as well as for ages 4–9 and 10–12, and 13–19 years, separately.
Our search strategy identified 9135 unique individuals between the ages of 4 and 19 years who had A1C level measures in our medical records. We excluded 3433 individuals due to height or weight values that were either missing (n=3409) or implausible (n=24). The demographics of the analysed sample of 5702 can be seen in table 1. On average, participants were (mean±SD) 13.0±4.1 years of age, 53% female, 57% white and 85% non-Hispanic. The sample consisted of 134 (2%) underweight, 1718 (30%) healthy weight, 660 (12%) overweight and 3190 (56%) obese individuals. The average A1C level of all participants was 6.04%±1.8%, and 16% of participants (n=936) had an A1C ≥6.5%.
Continuous relationship between BMI percentile and A1C
Among all participants, there was a weak negative correlation between BMI percentile and A1C (r=−0.20, P<0.0001). Linear regression revealed that for each 10% increment in BMI percentile A1C was 0.13% lower (95% CI −0.14 to –0.11)). The negative correlation occurred for participants in each of the three groups; ages 4–9 years (r=−0.22; β = −0.12 (95% CI −0.16 to –0.09)), ages 10–12 years (r=−0.27; β = −0.15 (95% CI −0.18 to –0.12)) and ages 13–19 years (r=−0.17; β = −0.12 (95% CI −0.14 to –0.10)).
Weight category risk of diabetes mellitus (A1C ≥6.5%)
As seen in table 2 and figure 1, in all age groups, those in the obese weight category had the lowest risk for an A1C ≥6.5%, and those in the healthy weight category had the highest risk for an A1C ≥6.5%. Children in the healthy weight category had 8.2 times (95% CI 5.3 to 12.7) the risk of A1C ≥6.5% than those in the obese category. In pre-teens, the relative risk (RR) was 13 (95% CI 8.5 to 20.0), and in teenagers the RR was 3.9 (95% CI=3.3 to 4.7).
These RR differences are especially notable given the prevalence of A1C ≥6.5% in our cohort. The absolute risk of an A1C ≥6.5% in healthy weight pre-teens was 38.9%, whereas in obese pre-teens the absolute risk of an A1C ≥6.5% was 3.0%. Consequently, the risk difference (RD) was 35.9% (number needed to treat=1/RD=2.8), suggesting that only three healthy weight pre-teens would need to become obese to prevent one case of diabetes. The n umber needed to treat (from healthy to obese) in children was 3.8, and in the teenagers was 4.2.
Risk of elevated A1C levels (≥8%) among those with diabetes mellitus
As seen in table 3, among those with diabetes, overall 77% had an A1C level ≥8%, signifying poor control. This percentage was generally similar between the weight categories with the exception of the underweight subjects who had a higher percentage with poor control in the child and teenage groups (RR=1.3 and 1.4, respectively).
In this study of 5702 children and adolescents, we found that obesity was negatively associated with A1C levels and the risk of diabetes, that is, the higher the BMI percentile, the lower the risk of A1C ≥6.5%. In our cohort, subjects who were of a ‘healthy weight’ had the highest risk of diabetes. Among children, pre-teens and teenagers, a healthy weight was associated with a 4–13 times higher risk of diabetes compared with those who were obese.
Since obesity is known to be a risk factor for diabetes, and yet in our study appeared protective, some would call our results ‘paradoxical’. However, our results are simply a textbook example of catastrophic selection bias. Consistent with the common definition of ‘selection’, we selected our participants based on a criterion (having an A1C drawn) that led to a biased sample. Selection bias can occur at various points in a study’s design or analysis when there is conditioning on a common effect of two variables, one of which is the exposure, or a variable associated with the exposure, and the other of which is the outcome, or a variable associated with the outcome.4 The causal inference literature refers to the common effect of two variables as a ‘collider’; hence, the synonymous term ‘collider stratification bias’ for this phenomenon.6 The name derives from the causal structure when two or more variables ‘collide’ at a particular node (variable) in a directed acyclic graph (DAG). In our study, obesity and illness ‘collide’ at the node representing a laboratory test for A1C, thus conditioning (via restriction) on having an A1C test leads them to be inversely associated in the study sample, even if they are unassociated/independent in the general population.6 Participants in our study were only included in the analytic cohort if their clinicians determined that a blood test for A1C level should be obtained, meaning that they had some type of clinical indication to warrant such a test. Many clinicians draw an A1C level when evaluating obese children and adolescents because recommendations include the consideration of testing for diabetes.19 Clinicians also draw an A1C level when children have symptoms of diabetes and appear ill, regardless of bodyweight status. This is the key to understanding the selection bias4–6: ordering the A1C is an effect of either being obese or appearing ill with symptoms and signs of diabetes.
We have drawn out a DAG representing our study design (see figure 2A). Since all subjects in our study had an A1C drawn, the analysis is conditioned on having had an A1C test by restricting the study population. A box around the A1C node represents this conditioning. Conditioning on having an A1C test induces an inverse and non-causal association between ‘obesity’ and ‘appearing ill’. We have used ‘appearing ill’ as an example of an unmeasured common cause of having an A1C test drawn and diabetes. Future studies could attempt to measure and then adjust for variables that are common causes of having A1C test drawn and diabetes, such as child thirst or urinary output. It is important to note that the simplified DAG presented in figure 2A and the structural diagram in figure 2Bb do not include potential confounders of the exposure–outcome relationship. For additional DAGs depicting various study design structures representing selection bias, we refer readers to other publications.4 20 Since having symptoms and signs of diabetes is a much stronger risk factor for diabetes than obesity, in our study design, obesity paradoxically appears protective.
Our study had all of the appearances of a high-quality medical publication. It was conducted at a major academic medical setting and abided by all of the tenets of scientific guidelines. We have attached our Strengthening the Reporting of Observational Studies in Epidemiology checklist as an addendum. Consider the apparent strengths of our study; we had a large sample size with 5702 different individuals. Thus, our findings, while wrong, are unlikely to be due to chance. The subjects were from a diverse population of urban and rural settings seen in the clinical setting. Therefore, we could tout that our results are likely generalisable to other clinic populations of children and adolescents. Finally, the P values were really, really small (<0.0001), and so many readers may think that there must be some truth in the findings.
Since paediatric obesity is clearly associated with decreased insulin sensitivity and a higher risk for chronic diseases, including diabetes mellitus,21–23 we could write a discussion section filled with headline-grabbing quotes describing our shocking results. Imagine news outlets getting hold of ‘Healthy weight is a risk factor for diabetes’. Since the risk (probability) differences might be attributed to causal effects, then, given the absolute RDs of 25.9%, 35.9% and 23.6% in the children, pre-teens and teenagers, respectively, industries selling high caloric foods and beverages might put together slick presentations demonstrating that clinicians need to help only 3–4 healthy weight youth become obese to prevent a case of diabetes; a huge return on investment.
We could write a section on our study’s limitations, with a few perfunctory comments such as the fact that heights and weights were performed and recorded by clinic personnel and subject to routine mismeasurements. Also, we might include the standard comments that BMI is an imperfect measure of obesity. Yet all of these strengths and limitations are trumped by the selection bias. Given the selection bias, there are no strengths that should make us more confident, nor limitations that should make us more uncertain about the results. As explained by others,24 selection bias does not lead to a mere limitation of generalisability, but rather affects validity because the associations in the sample do not have a causal interpretation. In the present example, selection bias was unavoidable as only a selected group of participants had an A1C test. Had we evaluated every child seen at our clinics between the ages of 4 and 19 years, there would have been no selection bias. Furthermore, there are analytic techniques to account for differential selection, such as inverse probability weighting, that could have been used to adjust for this form of bias.25
Unfortunately, selection bias has rarely proved a barrier to publication. There is an ever-increasing body of literature on studies that are structurally equivalent to the design of our study, whereby participants are restricted to an effect of the exposure and the results are labelled as a ‘paradox’. In our study, the non-obese subjects had another cause of their diabetes, that is, lack of insulin production. In the studies of patients with other ramifications of obesity,7–10 the patients who were not obese, by definition, had other causes for their illness. For example, heart failure can be due to genetic issues, infectious cardiomyopathy or cancer. If these other causes are more dangerous than obesity, then those who contracted heart failure due to obesity appear protected. Examples of selection bias have also appeared in other medical specialties. In the obstetrical literature, causes of small or pre-term babies (maternal smoking and pre-eclampsia) paradoxically appear protective when observing only babies below a certain weight or gestational age.2 Lack of understanding selection bias sets back advancements in medical science. For example, just a few decades ago, authors struggled in vain to provide a physiological explanation for the positive effects of maternal smoking on newborn health.20 26 Now, due to the articles claiming an obesity paradox, researchers are starting to question the standard the clinical recommendation for obese subjects with heart failure to lose weight.8 27
Of course, with diabetes there are considered to be two types, with type 1 traditionally thought to occur in the young and thinly built, and type 2 understood to be more common in the older and obese. In fact, there is much more overlap than those classic divisions suggest. A database of >5500 adolescents with type 1 diabetes found that 23% were overweight and 13% were obese.28 Furthermore, in youth with type 2 diabetes, the mean age of onset is 13.5 years.29 The one stereotypical feature that seems to prevail is that few, if any, type 2 diabetics are lean. The fact that obesity is, on a population basis, more common in type 2 than in type 1 diabetes creates the observed inverse correlation with illness in our study when one selects only those considered at risk by their clinician.
Were our example not so absurd and contrary to existing knowledge, it might pass for a credible finding, and children might be recommended for fast-food gavage as a means to stave off diabetes. While paediatric obesity, similar to smoking, as a cause of disease prevention appears absurd, adult obesity as a health benefit seems more plausible. Obesity is associated with increased muscle mass and bone strength. Yet, the structure of the selection bias is the same in our study as in studies reporting that adult obesity is protective against outcomes that it causes. Risk factors can appear paradoxically protective if researchers restrict their studies to selected populations and disregard the bias that affects all of the results. The punchline here is that selected samples can contain non-causal associations that reverse the true causal relations present in the general population. Researchers who ignore this simple and repeatable fact will generate harmful misinformation that is no joke.
What is already known on this subject
Selection bias can occur when inclusion criteria are based on a consequence of exposure and may produce results that appear in the opposite direction of the true causal effects.
The medical literature is full of examples where selection bias produces counterintuitive results that are labelled a ‘paradox’ with a call for more research.
Diabetes in children and adolescents is a result of either insulin resistance (caused by obesity) or lack of insulin production.
What this study adds
A clear example where selection bias produces results that are counterintuitive and nonsensical.
Not a whole lot to our understanding of the link between paediatric obesity and diabetes.
A good strategy for generating headline-grabbing results: study only those with a defined outcome and then look for a cause of the outcome that appears ‘paradoxically’ to be protective.
The authors thank Samantha Carlson, MPH for assistance with the statistical analysis. Support was provided by Research Services, Department of Family Medicine and Community Health, Medical School, University of Minnesota.
Contributors All authors contributed to the concept and design of the study, as well as the analysis and manuscript writing.
Funding HRB received funding from the Canadian Institute of Health Research Banting Postdoctoral Fellowship Award.
Competing interests None declared.
Ethics approval Our study involved deidentified analysis of pre-existing data and was determined by the University of Minnesota’s Institutional Review Board to be exempt for the need for further review.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.