METHOD

Sample selection

To provide demographic, socioeconomic, and academic diversity among the study sample, 13 public (state supported) high schools were chosen from a population of 114 “comprehensive” and “vocational” New York City high schools. The schools were stratified by type (vocational or comprehensive) and degree of absenteeism and then randomly selected within these strata. Seven “comprehensive” high schools were included; these schools are typically local high schools without special entrance criteria. Two additional “educational options” schools requiring special selection procedures were included in the sample as were four “vocational high schools”, which train students for careers in particular industries. Half of the schools within each category were selected because they had been identified as having particularly high rates of absenteeism. Tenth graders were selected because, in the United States, the majority are 15 years of age, and thus too young for legal self exemption, or dropping out without parental consent, from school. Human subjects approval was obtained both from New York University and from the New York City Board of Education. A pilot study was conducted in two additional schools in the autumn, 1997, and full field operations were put in place in spring, 1998.

Survey design

Risk behaviours considered in this study include drug and alcohol use, absenteeism, academic achievement, sexual activity, and weapons possession. Questions were drawn from several previous US studies of adolescent risk behaviour, including the CDC’s Youth Risk Behaviour Survey, University of Michigan’s Monitoring the Future Study, the New York State Office of Alcoholism and Substance Abuse Services’ Study of Youth-At-Risk, and the Robert Wood Johnson Foundation funded evaluation of the New York City Public Schools HIV/AIDS Education Program. All selected questions were validated and field tested, and were written for high school students. Wording for the relevant study questions is provided in box 1.

Research design

The data presented in this paper were gathered as part of a larger study examining the impact of student absenteeism on estimates of youth risk behaviour. The larger study used a four stage data collection strategy, in which each stage represented an increased level of intensity and associated resources. By virtue of the study design, those who were surveyed in the later stages were more likely to be absent from school or to frequently skip classes than those students surveyed in the earlier stages.¹⁵ The four stages were implemented sequentially without overlap; students surveyed in stage 1, for example, were not included in stages 2, 3, or 4. Both stages 1 and 2 were classroom based, and replicate the survey methods used in previous studies on adolescent risk taking behaviour. Stages 3 and 4, took place outside of the classroom (usually an office or library), incorporated financial incentives, and represented a substantial departure from previous research in the field. For maximum comparability to survey approaches typically used in the study of adolescent risk taking behaviour, only data on students surveyed in stages 1 and 2 are included in this paper. Findings concerning the comparison of stages 1 and 2 respondents with those of stages 3 and 4 are to be found in Guttmacher et al.¹⁵

Of note, despite the introduction of financial incentives (gift certificates to local stores) and the use of more intensive and costly efforts to reach students who were not in the classroom during the survey administration (for example, letters inviting participation in the survey were sent to the homes of all non-respondents), the increase in the number of student respondents was comparatively small. The initial sample included 2675 students; 1921 were interviewed in the classroom setting, 128 were interviewed as a result of the more intensive outreach efforts, and 626 students (23% of the original sample) were never interviewed. The findings from this earlier study suggest that the increased expense associated with more intensive methods of outreach for classroom based studies may not be justified by improved estimates as comparatively few additional students are surveyed. As classroom based surveys are typically undertaken to reach large numbers of students at comparatively low cost, the level of incentive that may be needed to attract frequently absent students is not probably affordable in most studies.

To differentiate respondents from non-respondents, surveys were kept confidential but not anonymous. Trained data collectors, who administered the survey, reaffirmed the confidential nature of the survey. As almost all research subjects were under the age of 18, both student and parental consent were required before survey administration. About 4% of parents/guardians requested that their child be exempted from the study.

The demographic characteristics of the final study sample closely approximate that of the New York City public high schools.⁹ Slightly more than half of the sample was female and 87% of the sample was between 15 and 16 years of age, which is the appropriate age for 10th grade students. There were approximately equal proportions of African-Americans, Latinos, and Caribbean-Americans, whereas Asians and white groups, combined, accounted for 15% of the students. Ninety four per cent of all respondents surveyed in either stage 1 or 2 (n=1854) are included in this paper.

Analysis

To understand the impact of excluding low response rate schools, schools were categorised as having either high or low response rates according to the proportion of students who were surveyed by the end of stage 2. Schools achieving a 70% or greater response rate were categorised as “high response schools,” whereas those not achieving a 70% response rate by the second stage of follow up were categorised as “low response schools.” Nine schools met the criterion for high response (n=1397) and four schools met the criterion for low response (n=457). The cut off points were chosen to reflect the inclusion criterion used in some other adolescent health surveys used in the United States. To keep these analyses comparable to those used in other studies, only students from stage 1 were included from schools where a 70% response was achieved after stage 1. Students from stages 1 and 2 were included from the other schools. Difference of proportions tests (χ² and t tests, as appropriate) were used to test for the statistical significance of the differences between students in low and high response rate schools. Data were then weighted on the basis of student attendance; the weighting protocol assumes that the probability of a student being surveyed is inversely related to the proportion of days absent as reported by the student. Weighted estimates of risk behaviour—for the inclusion and exclusion groups—were compared.

RESULTS

Sample demographics and risk behaviours

In table 1, we see that students surveyed in low response rate schools are more likely to be female (p<0.001), African-American (p<0.001), and from single parent or non-parent households (p<0.001). As would be expected, students in high attendance schools are far less likely to report non-legitimate absences or class skipping; for example 74% of students in high attendance schools report never skipping classes, as compared with 30% in low attendance schools (p<0.001). Students in low attendance schools also report far weaker student performance. Whereas 39% of students in high attendance schools report receiving mostly grades A and B, only 29% of students in low attendance schools report such achievement (p=0.001).

There are also variations in risk behaviour across school groups. In particular, reports of cigarette use, marijuana use, and HIV risk behaviour are higher for students in low attendance schools. Only 15% of students in high attendance schools reported marijuana use in the past four weeks, while 21% of students in low attendance school reported such use (p=0.002). Similarly, 19% of youth in low response rate schools had smoked cigarettes in the previous four weeks, as compared with 14% in high response rate schools (p=0.020). Seventeen per cent of students in low response rate schools indicated that they engaged in behaviours that put them at high risk of HIV, as compared with only 13% in high response rate schools (p=0.025).

Despite these differences, limiting the sample to high response rate schools did not substantially change the estimates. When we compare the unweighted data for students in the high response schools, we find a marijuana use rate of 15% as compared with 16.5% among students in all schools. Once the data are weighted for individual student absences, we find very small but consistent increases in the estimates of risk behaviour. The estimate of marijuana use rises to 17% when students from high and low response rate schools are included and the data are weighted for individual absences. While these differences across samples and methods are not statistically significant, the direction is consistent; when low response rate schools are eliminated and when the data are not weighted for absences, there is a risk that estimates of risk behaviour will be too low.

In this particular case, the ratio of high to low response rate schools was 9:4, and the ratio of survey respondents from high to low response rate schools was about 3:1. If, however, the split between low and high response rate schools had been more balanced, or the proportion of respondents from low response rate schools greater, the bias introduced by eliminating low response rate schools would have been greater.

DISCUSSION

Eliminating schools that have not achieved a 70% response rate does not seem to be helpful to estimating risk behaviours among youths. Norms of appropriate behaviour related to school attendance seem to differ from school to school and schools with lower rates of attendance include students who are more likely to engage in a number of risk behaviours. Any method that systematically rejects schools with high rates of absenteeism is also systematically rejecting those schools with highest concentrations of risk behaviour.

In school systems where there is considerable absenteeism, weighting the data on the basis of individual self reported absences seems to improve the overall estimates of risk behaviour and should be routinely used in school based survey research. Currently many studies, including the national YRBS in the United States and the international HBSC, do not follow this protocol and, as a result, they may be underestimating certain behaviours. In addition, the practice of eliminating low response rate schools does not seem to have “pay off” and could result in an underestimate of risk behaviours, especially in those school systems where the differences between low and high attendance schools are large and the proportion of low attendance schools are great. As evidence suggests that students not surveyed in low attendance schools (that is, those not “captured” by the survey because of long term absence) are likely to engage in even more risk behaviours than their surveyed counterparts, this bias is compounded.^8,11,15 While on the surface it might seem that eliminating schools with poor response rates would strengthen the estimates, we find no evidence of a beneficial effect and some suggestion of possible detriment. Most notably, the youth in these schools, who seem to be the ones with the greatest degree of academic problems, are of particular importance to the study of risk behaviour and should not be systematically ignored by the field.

Survey researchers interested in adolescent risk behaviour are faced with a difficult situation. Absenteeism and risk behaviour seem to correlate; schools and communities with higher rates of absenteeism also tend to experience higher rates of risk taking behaviour. Yet, in those localities where rates of absenteeism are greatest, classroom based surveys may be of limited value because they do not include students who are rarely in the classroom. Various attempts to reduce this potential source of bias by intensifying the outreach effort have proved weak or ineffective.¹⁵ Furthermore, these intensive efforts can be expensive and time consuming, thus reducing the cost-saving benefits of classroom based studies. In non-school based studies of adolescent risk behaviour, more intensive efforts to track students have been effective, albeit expensive and time consuming.¹⁷ Of course, the use of more intensive efforts (such as financial incentives) may further bias the study pool if they are of greater appeal to one group than another. As researchers attempt to sort out these competing risks of bias, policy makers and officials in adolescent serving agencies must recognise that current estimates of risk behaviour, typically based on classroom surveys, may underestimate the extent of such behaviour among all adolescents. Programme development should recognise that adolescents who attend school erratically or sporadically may have different, and greater, needs for intervention than those who are included in classroom based surveys. If our primary interest is the study of those adolescents who engage in the most risky behaviours, simple classroom based surveys may not be the most appropriate method.