Background Response rates of national health surveys are decreasing, which potentially can bias obtained prevalence estimates. The purpose of this study is to evaluate the extent to which non-response impacts the representativeness of the 2000 Behavioral Risk Factor Surveillance System (BRFSS) sample compared to the 2000 Decennial Census.
Methods The 2000 BRFSS had a median response rate of 48%, while the 2000 Decennial Census had a response rate of 67%. Representativeness of the BRFSS sample was evaluated on gender, race, ethnicity, age, household income and marital status. Prevalence of each factor in the BRFSS was compared to the prevalence found in the US Census on both the state and county levels. Prevalence differences between the BRFSS and Census were calculated and their association with response rates was evaluated using robust OLS regression and polytomous logistic regression. The relationship between prevalence differences and other survey design elements, such as data collection procedure and sampling fraction, was also explored.
Results The BRFSS prevalence estimates diverged from the Census estimates on several sociodemographic factors even after adjustment for non-response/non-coverage. This was found on both the state and county levels; however, smaller absolute differences between the BRFSS and Census prevalence estimates were found for factors included in the non-response/non-coverage adjustment weight. Lower response rates (<40%) were associated with the under-representation of racial/ethnic minorities, women and younger individuals in the BRFSS survey.
Conclusion Future research should examine alternative approaches to increase response rate (eg, mixed mode) and to adjust for non-response (eg, multiple imputation).
- Health surveys
- behavioural risk factor surveillance system
- surveillance SI
- surveys me
Statistics from Altmetric.com
The Behavioral Risk Factor Surveillance System (BRFSS) is a national random-digit-dial (RDD) health survey sponsored by the Centers for Disease Control and Prevention (CDC). Over the past decade, this annual survey's response rates have declined both on the state and national levels.1 Response rates are generally used as indicators of potential bias as low response rates increase the possibility that survey error will bias results.2 In 2000, the median state response rate for the BRFSS was 48.9% (range: 28.8%–71.8%).3
The CDC attempts to adjust for non-response and non-coverage (NR/NC) with a weight based on the age, gender and racial distribution of the targeted population for a state or region. The underlying assumption with this method is that non-responders are missing completely at random—that is, non-responders would answer all survey questions identically to responders in the same age, sex and racial category. Yet, the amount of bias introduced by non-response depends on both the proportion of the sample that fails to respond and the extent to which non-responders systematically differ from the entire population. If characteristics of non-responders are distinct from the actual population, survey prevalence estimates may be underestimated or overestimated.
The generalisability of BRFSS samples could vary by state and by year due to differences in response rate. Non-responders are more likely to be younger, race other than white, less educated, non-English speakers and unmarried.4 5 Thus, there may be greater differences in sociodemographic characteristics—beyond age, race and gender—between the BRFSS sample and the actual state population for states with low response rates.
The purpose of this study was to evaluate the extent to which characteristics of the BRFSS sample diverged from the Census on both the state and county levels, and to quantify the extent to which NR/NC adjustment methods produced a representative sample on both factors included and not included in the NR/NC adjustment weight. If the missing completely at random assumption holds, we would expect the NR/NC weight to adequately adjust for non-response on all factors. Additionally, we explored the association of survey design elements on the magnitude and direction of the difference between BRFSS and Census estimates.
The 2000 Decennial US Census provided information on state and county level sociodemographic factors among the non-institutionalised population.6 Census data often serve as a gold standard7 and its mixed mode approach generally results in higher response rates and more representative samples than telephone surveys.8 The response rate for the 2000 Decennial Census was 67%.9
We also used the 2000 BRFSS, which samples civilian, non-institutionalised persons 18 years and older using RDD methods from a probability sample of households with landline telephones.10 11 It is sponsored by the CDC and implemented by individual health departments in all 50 states, the District of Columbia (DC) and US territories.
Our dependent variables were indicators of selection bias measured by absolute differences in prevalence estimates between the BRFSS and Census. We selected factors related to non-response that were ascertained with comparable question wording and response categories in both the BRFSS and Census.4 12 We selected factors included in the BRFSS NR/NC adjustment weight (age, gender, race and ethnicity) and not included in the weight (income and marital status). For age, we estimated the proportion of individuals aged 18–34 years (35 years is the median age in the US).13 Race was represented by per cent white, gender by per cent women, ethnicity by per cent Hispanic, income by per cent with annual household income <$15 000 and marital status by per cent married.
We considered several survey design elements as potential correlates of absolute differences between BRFSS and Census prevalence estimates. Our main correlate was response rate. Following conventions set by the Council of American Survey Research Organizations (CASRO) and equations provided on the BRFSS website,2 we calculated response rates using disposition codes obtained directly from the CDC (personal communication with Michael Link, October 2006).
Other correlates included: (1) data collection procedure—in 2000, 13 states collected in-house and 37 states and DC contracted out data collection (personal communication with state BRFSS coordinators, December 2006); (2) state-level sampling fraction was created by dividing BRFSS sample size by Census sample size; and (3) state-level average number of call attempts before respondent completed the BRFSS questionnaire. The CDC recommends at least 15 call attempts over five separate calling occasions to complete an interview.
Census region was considered a confounder because regional differences exist in survey non-response and sociodemographic profiles14 15; West North Central served as the reference group since this region had the highest response rate. Also, we controlled for state-level landline telephone coverage (from 2000 Census) since the BRFSS NR/NC adjustment weight also corrected for this variable.11
Analyses were conducted on the state and county levels. State analyses included all 50 states and DC. County level analyses were important due to an increasing need by local health departments for county-specific information16 and the BRFSS is often used to produce county-level estimates of risk factors and health behaviours when sample size allows.17 Yet, the BRFSS NR/NC adjustment was not developed for county level of analysis. County level analyses excluded Alaska, since it has no Federal Information Processing Standard (FIPS) county codes to link with Census data, and counties with <50 persons, the minimum sample size recommended by the CDC to produce stable estimates.18
BRFSS data were weighted to adjust for the complex survey design, such as the sampling of individuals within households within states, using SAS-Callable SUDAAN.19 We evaluated the extent and direction of non-response bias for both the state and county levels using two conceptually different outcomes: (1) absolute differences between BRFSS and Census prevalence estimates; and (2) differences categorised according to the direction of bias.
First, we determined the extent to which prevalence estimates of sociodemographic factors derived from the BRFSS differed from the Census. For each factor, we calculated the prevalence difference between the BRFSS and Census estimates to determine if bias was present. A zero difference indicated no bias present and a positive or negative value indicated some degree of bias. To show the effectiveness of the NR/NC adjustment technique in eliminating non-response bias, we estimated and compared prevalence differences between BRFSS and Census: first, using BRFSS estimates not adjusted for NR/NC (BRFSS-NA) and then BRFSS estimates adjusted for NR/NC (BRFSS-A). Since the prevalence estimates were normally distributed, we used paired t tests to determine if differences between the Census and the two BRFSS estimates (BRFSS-NA and BRFSS-A) were significantly different at p<0.05. We hypothesised that larger absolute differences between the BRFSS and Census prevalence estimates would be observed on all factors pre-NR/NC adjustment versus post-adjustment. However, adjustment for NR/NC would attenuate differences for factors comprising the NR/NC adjustment weight, while differences would persist for factors not included in the weight. For each factor, we reported median values and interquartile ranges (IQR).
Second, we evaluated the extent to which response rate and other survey design elements explained the absolute differences between the BRFSS and Census by using robust ordinary least squared regression. Separate models were constructed for each of the six sociodemographic prevalence differences and on both the state and county levels. We constructed crude and adjusted models. The adjusted models included the four survey-related variables and two potential confounders. Correlations between independent variables were examined to rule out multicollinearity. We hypothesised a negative relationship between response rates and prevalence differences for factors not used in the adjustment weight for NR/NC. We posited that smaller absolute differences between BRFSS and Census prevalence estimates would be observed in states that collected data in-house compared to states that used contractors since interviewer retention is higher in-house,20 states with higher sampling fractions since they capture a larger proportion of the total population and states with a larger number of call attempts.
For county-level regression models, we adjusted for clustering of counties within states.21 We had increased power with county as the unit of analysis (n=∼740) due to the larger sample size compared to state-level analyses (n=51). On the county level, we hypothesised that absolute differences in prevalence estimates derived from BRFSS and Census would be strongly related to response rate for all factors since the NR/NC technique is primarily based on the state level.
Third, we implemented an analytic approach to determine the impact of response rate and other survey design elements on the direction of bias. County was the unit of analysis due to the larger sample size. A three-level outcome variable was created based on absolute differences in prevalence between BRFSS and Census estimates: BRFSS overestimating Census (BRFSS-A prevalence >2.5% points higher than Census prevalence), BRFSS underestimating Census (BRFSS-A >2.5% points lower than Census) and BRFSS approximating Census (BRFSS-A within 2.5% points of Census).
We estimated adjusted ORs and 95% confidence limits (CLs) for each sociodemographic prevalence difference using robust polytomous logistic regression with BRFSS overestimating Census serving as the reference group. For interpretability, we classified the continuous correlates from our prior analyses as categorical. Response rates were classified as <40%, 40–60% and >60% according to CDC quality control guideline minimum acceptable CASRO response rate of 40% and the Office of Management and Budget standard for generalisable surveys of 60%.22 23 We performed a median split for sampling fraction (≤0.14% and >0.14%) and average number of call attempts (>5 attempts and ≤5 call attempts). All descriptive and regression analyses were conducted in Stata.24
Range of sociodemographic prevalence estimates: BRFSS versus Census
Table 1 displays the range of prevalence estimates for six sociodemographic factors derived from the Census, from the BRFSS-NA and from the BRFSS-A on both the state and county levels. On the state level, we observed minimum differences in prevalence ranges for factors used to adjust for NR/NC between the Census and the two BRFSS estimations. The Census range for ethnicity was 0.6–38.8%, the BRFSS-NA was 0.8–41.4% and the BRFSS-A was 0.7–42.1%. Adjustment for NR/NC did correct the BRFSS prevalence range for gender (Census: 49.1–54.2%; BRFSS-A: 47.9–54.0%; BRFSS-NA: 48.4–66.9%). For the two factors not used to adjust for NR/NC, their prevalence ranges for BRFSS-NA were more similar to BRFSS-A than to the Census. The range of prevalence estimates for income was 5.3–18.4% for BRFSS-NA and 5.5–18.6% for BRFSS-A, while the Census range showed greater prevalence of low-income individuals (10.6–25.4%).
On the county-level, the Census minimum prevalence for age was 16.6%, while the minimum prevalence for BRFSS-NA was 8.7% and BRFSS-A was 9.8%. Adjustment for NR/NC overestimated gender (69.7%) and age (71.9%) compared to the Census (56.0% and 62.4%, respectively). Ranges of prevalence estimates for factors not included in the NR/NC weight were more comparable between BRFSS-A and BRFSS-NA than to the Census. Prevalence of income ranged from 0.0–32.8% for BRFSS-NA and from 0.0–36.3% for BRFSS-A compared to the Census (3.1–43.0%).
Maximum differences in prevalence: BRFSS versus census
Table 2 shows the impact of the BRFSS adjustment technique on reducing bias. On the state level, absolute differences between the BRFSS and Census prior to NR/NC adjustment ranged from 6.6 (ethnicity) to 20.6 (race). For those factors included in the NR/NC weight, adjustment decreased the absolute difference for gender (from 14.4 to 0.1) and age (from 5.0 to 0.1). However, racial/ethnic differences were not eliminated (race: 20.6 to 18.4; ethnicity: 6.6 to 4.7). In terms of variability, the medians and IQRs remained unchanged for age, race and ethnicity. On the state level, prevalence estimated using BRFSS-NA was significantly different from the Census for all factors, and prevalence estimated from the BRFSS-A was significantly different from the Census for race, ethnicity and income (p<0.05)—two factors included in the NR/NC adjustment weight and one factor not included in the weight.
Absolute differences in prevalence for factors not included in the NR/NC weight were 8.0 (marital status) and 9.8 (income); adjustment for NR/NC did not appreciably alter the size or variability of the absolute prevalence differences for these two factors. For income, the median changed from 6.2 to 6.0 and the IQR changed from 3.2 to 3.0 with adjustment.
Prevalence differences between the BRFSS and Census were larger on the county level. For factors included in the NR/NC adjustment, the maximum absolute differences were all large prior to adjustment (gender: 24.8; race: 32.1; age; 21.4; ethnicity: 16.9) and not appreciably altered with adjustment (18.2, 29.9, 26.9 and 18.3, respectively); the medians and IQRs also remained unchanged with NR/NC adjustment. A similar pattern was observed for factors not included in the NR/NC weight. On the county level, both the BRFSS-NA prevalence estimates and BRFSS-A prevalence estimates were significantly different from the Census for all factors (p<0.05).
Results from OLS regression analyses
After controlling for Census region, telephone coverage and other survey design elements, response rate was not associated with differences in prevalence between BRFSS and Census for factors included in the NR/NC adjustment weight (gender, race and age) (Table 3). In-house data collection (vs contracted) was associated with a 3.06 (95% CL 0.14 to 5.98) larger prevalence difference between the BRFSS and Census for race. For each additional call attempt, the prevalence difference for gender decreased by 0.29 percentage points (95% CL −0.48 to −0.10).
For county-level analyses, no association was found between response rate and prevalence differences for gender, race and age. In-house data collection was associated with greater differences in prevalence between the BRFSS and Census for gender (β=3.13; 95% CL 0.25 to 6.01).
On state and county levels, response rate was not associated with prevalence differences between the BRFSS and Census for factors not included in the NR/NC adjustment weight (income and marital status). However, increased sampling fraction was associated with a decrease in the difference between BRFSS and Census for prevalence of income on the state level (β= −3.42; 95% CL −6.79 to −0.05).
Impact of response rate and survey design elements on direction of bias
Table 4 shows that relative to counties with BRFSS response rates of ≥60%, counties with response rates <40% had approximately twice the odds of underestimating (vs overestimating) women (AOR 2.11, 95% CL 1.10 to 4.09) and individuals aged 18–34 years (AOR 2.16, 95% CL 1.09 to 4.27). Counties with response rates of 40–60%, compared to counties with response rates of ≥60%, had 1.32 greater odds of underestimating Hispanics (95% CL 1.07 to 1.64). White race was less likely to be underestimated than overestimated with response rates <40% (AOR 0.31, 95% CL 0.11 to 0.88) and 40 to <60% (AOR 0.76, 95% CL 0.62 to 0.94) compared to high response rates.
In this paper, we sought to evaluate the role of non-response bias in producing compositional differences between the BRFSS sample and the Census. We failed to find strong linear associations between response rate and compositional differences at either the state or county levels. However, we did find that counties with very low response rates (<40%) were more likely to underestimate hard-to-reach populations—for example, racial/ethnic minorities and younger individuals—even after NR/NC adjustment. In 2000, 25% of states and 16% of counties fell below the BRFSS minimum response rate and, thus, may have underestimated populations marginalised in society. This is concerning as prevalence estimates are routinely used to evaluate progress towards Healthy People 2010 goals and to target resources. Using BRFSS data from states or counties with response rates <40% may overstate progress towards achieving Healthy People goals since marginalised populations are likely under-represented.
Before adjustment for NR/NC, there were large differences in prevalence between the BRFSS and Census for all sociodemographic factors. After applying the adjustment weight, the magnitude of difference and variability for factors included in the weight, such as gender and age, were largely reduced. However, adjustment was less effective in eliminating large differences in prevalence for race, ethnicity, income and all factors on the county level. This evidence suggests that the missing completely at random assumption, which is the basis for the NR/NC weight, may not be completely valid since differences persisted after adjustment for variables beyond age and gender. It was effective in minimising differences for factors included in the NR/NC adjustment weight at the state level and likely also minimised differences for factors correlated with the NR/NC adjustment factors, but less effective for factors not included in the weight or presented at the county level. Caution is warranted when interpreting county-level BRFSS prevalence of factors not included in the NR/NC weight or at the county level as they can be as much as 20% points different from Census estimates.
Using a weighting technique that assumes missing completely at random was effective at varying degrees across our selection of factors. Other techniques that have been developed to correct for NR/NC may be more effective at adjusting for non-response for factors beyond those included in the NR/NC adjustment weight, such as multiple imputation methods that consider unit response as an extreme form of item non-response,25 Bayesian methods that use information about responders to predict prevalence estimates26 and propensity scores that link unit non-response to item non-response.27 Assumptions about missing data (ie, non-responders) are also made with these alternative methods and their effectiveness in mitigating non-response bias must be explored.
We were unable to isolate any modifiable survey design factor consistently associated with prevalence differences between the BRFSS and Census. The multivariable OLS regression models explained 35–50% of the variance in prevalence differences. There are potentially other factors not considered in this study. Interviewer characteristics and style, such as gender and years of experience, are correlated with response rate.28 However, information on these variables was not available to include in the analyses.
Our findings must be considered with a few caveats in mind. First, as mentioned above, the 2000 Census response rate was only 67%.9 Thus, it is important to note that Census data are also subject to non-response bias even though the response rate for the 2000 Census was approximately 20% points higher than the BRFSS' median response rate in 2000. In the early 2000s, the Census Bureau conducted analyses to explore the possibility of adjusting for over and under-counts,29 but decided not to due to concerns surrounding technical limitations of the adjustment methodology.
We were also limited by the number of variables available in both BRFSS and Census with comparable definitions. The variables available in both datasets were sociodemographic and it is unknown how non-response bias influences health-related variables. We analysed variables not included in the NR/NC adjustment weight; however, if these variables were positively correlated with variables comprising the weight then NR/NC bias would be minimised for them as well. We were unable to evaluate these relationships since Census data are not available on the individual level.
Additionally, counties with small samples (<50) were excluded from our analyses because they produced unstable estimates. The generalisability of our findings to such counties is questionable. No survey design elements were consistently related to differences in prevalence between the BRFSS and Census across all sociodemographic factors. Since multiple statistical hypotheses were tested, it is possible that at least one of the significant associations found in our analyses was due to chance; however, the probability is small.30 Lastly, while we evaluated the impact of non-response bias on the estimation of prevalence, the extent to which non-response bias affects inter-relationships between BRFSS variables is unknown.
The BRFSS provides national and state estimates of health behaviours and preventive practices to track progress on state and national Healthy People 2010 goals. Given the persistent decline of response rates, prevalence estimates obtained from the BRFSS should be interpreted with caution especially at the county level. Our study found that the BRFSS sample diverged from the Census state and county composition on most sociodemographic factors, even after NR/NC adjustment, and that lower response rates (<40%) were associated with the under-representation of racial/ethnic minorities, women and younger individuals in the BRFSS survey. Alternative approaches should be considered to increase response rate and/or to adjust for non-response.
What is already known on this subject
Response rates of health surveys administered over the telephone have been decreasing, which may result in biased estimates. This is concerning since surveys like the United States' Behavioural Risk Factor Surveillance System are used to track Healthy People 2010 goals at the national, state and local levels.
What does this study add
Including a non-response/non-coverage adjustment factor in the survey weight does not necessarily eliminate the bias associated with non-response. Prevalence for factors not included in the weight and prevalence at local levels (eg, counties) should be interpreted with caution since the estimates may remain biased even after adjustment for non-response/non-coverage.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.