METHODS

The Young Adults in Community Study (YAICS) is part of the Children in Community Study (CICS).³⁵ The purpose of the CICS was to document the developmental course of a group of children who were similar to the general US population in socioeconomic status, family structure, and urban/rural status and to identify factors related to the onset and persistence of psychiatric disorders. The YAICS focuses specifically on the period of transition from adolescence to young adulthood. The YAICS was reviewed and approved by the University of Washington Institutional Review Board.

Study sample

The CICS study sample was selected in 1975, using a four stage sequential procedure to obtain a probability area sampling of representative families in Albany and Saratoga counties in upstate New York with children between the ages of 1 and 10 years. One child was randomly selected from each qualified household. Sampling methods are described fully by Kogan et al.³⁶ Completed interviews were obtained from 976 of the 1141 qualified households, yielding an initial response of 86%. The first follow up (wave 2) was carried out in 1983 when 74% of the original respondents were re-interviewed. Five children had died, 10 were located in areas too distant or isolated for follow up, and 96 families refused participation or had scheduling problems that precluded conducting interviews within the designated follow up window. The families who were lost to follow up tended to have the youngest children (ages 1–4 years in 1975) and to live in areas of urban poverty. To replace the segment of the original sample that had been disproportionately lost to follow up, 54 additional families with children in the youngest age range and that lived in poor urban neighbourhoods were recruited using the same enumeration and sampling procedures as were used in 1975. With this supplement, the wave 2 study cohort was closely representative of children in the geographical areas sampled, as confirmed by comparison with the 1980 census.³⁷ During the second follow up (wave 3), conducted in 1985–86, 96% of the families interviewed in 1983 were re-interviewed.

Participants in the YAICS were the 181 youths from the CICS study sample who had not yet reached the age of 18 at the time of their 1983 interview and who were 18 years of age or older at the time of their 1985–86 interview, an average of 2.5 years later. These youth were at the upper end of the age range (8–10 years old) in the original study cohort. An estimated 20% attrition occurred in this age subgroup between waves 1 and 2, and an additional 3% attrition occurred between waves 2 and 3.³⁸

Study methods for the CICS and the YAICS have been detailed in previous publications.^14,35,36 The wave 2 and 3 study protocols involved pairs of trained lay interviewers conducting in person interviews with children and their mothers in family homes.

Classification of psychiatric disorder

Methods for measuring three dimensions of adolescent psychiatric disorder—diagnosis, symptom score, and level of adaptive functioning—are described below. Psychiatric disorder was assessed based on data collected in 1983, when the study participants ranged in age from 13.8–17.9 years (median 16.7).

Diagnosis

The first step in assigning a psychiatric diagnosis was to pool symptom endorsements from responses obtained from administering both child and parent versions of the Diagnostic Interview Schedule for Children (DISC-Y, DISC-P).^39,40 The next step was to determine whether the adolescent met DSM-III diagnostic criteria for disruptive (attention deficit hyperactivity disorder, oppositional defiant disorder, and conduct disorder); anxiety (overanxious disorder, separation anxiety disorder, and social phobia); affective (major depressive disorder); or substance abuse disorders (alcohol, tobacco, marijuana, and other drug abuse), based on child or parent symptom endorsements. The field of child psychiatric epidemiology has determined that sensitivity is improved significantly by using the either/or approach.^41,42 Finally, because pooling positive symptom endorsements from either parent or child yielded a very high prevalence of “any diagnosis,” an additional criterion was applied. The final step in assigning a diagnosis was to determine whether the number of pooled symptoms for a given disorder was one or more standard deviations (SD) above the mean number of symptoms for that disorder within the entire study sample.⁴³ Adolescents who both met diagnostic criteria and whose number of symptoms was one or more SD above the mean were considered to have a psychiatric disorder.

This two stage algorithm for assigning psychiatric diagnosis has been demonstrated to have validity in studies where prevalence estimates resulting from their application were shown to be consistent with estimates from other studies, and estimates of prevalence for specific disorders in age and gender subgroups were shown to be similar to expected patterns. Childhood diagnoses established by these methods have demonstrated strong correlations to known prenatal, perinatal, and familial risk factors for disorder measured before diagnosis, and to functional status measured concurrently, as well as to known sequela of disorder.⁴³

Symptom score

To calculate a symptom score, counts of psychiatric symptoms and severity indicators endorsed by parent and/or child report were tallied for each of the 11 diagnosis specific symptom scales. Each scale was modified to have an equivalent range from 0 to 10 by subtracting the lowest score from every individual subject's score, dividing by the range, and multiplying by 10. The total symptom score was then calculated as the sum of the diagnosis specific symptom scores and ranged theoretically from 0 to 110. The disruptive symptom score was calculated as the sum of the three disruptive diagnosis specific symptom scales and ranged theoretically from 0 to 30.

Although the symptom count scale is based on the same information as the diagnostic scale, most of its variance comes from the subgroup without a diagnosis, while the diagnosis classification simply discriminates between the sample including that subgroup and the diagnosed subgroup.

Level of adaptive functioning

Measures of adaptive functioning were constructed with the intent to create analogues to a combination of two scales to reflect an individual's success or impairment in developmentally normative role functioning, the DSM-IV “Social and Occupational Functioning Assessment Scale”⁴⁴ and the “Children's Global Assessment Scale”.⁴⁵ Both of the functional assessment scales focus on how a person is doing at home, with peers, and at school or work. For both scales, youth are rated by clinicians who are instructed to consider such aspects as participation in hobbies or activities, difficulties with school/work, the presence of self doubts, the frequency of interpersonal conflict, and the amount of defiance shown in the home.

To construct the level of adaptive functioning analogue, seven scales from the Children in the Community Study were used. The seven scales with their number of items and internal consistency coefficients included: academic achievement (4, 0.61), general sociability (5, 0.56), and self esteem (9, 0.72), (constructed for the CICS by principal investigators Patricia Cohen and Judith Brook), interpersonal difficulties (5, 0.71),⁴⁶ resistance to maternal control (5, 0.88),⁴⁷ and social competence (4, 0.33) and participation in activities (20, 0.67).^48,49 The internal consistency coefficients for the scales that comprised the Adaptive Functioning Scale were generally good, with the exception of social competence. Despite its low coefficient, this scale was maintained, because social competence is a key component of adaptive functioning and is measured by a subscale of the Child Behavior Checklist (CBCL),⁵⁰ a widely used psychometric instrument.

Examination of the distributions of scores for each scale showed them all to be fairly normally distributed. For each of the seven subscales, study participants were coded as “1” if they were in the least functional quartile of the distribution and “0” if they were above the lowest quartile of functioning. The codes were then summed across each of the seven individual scales yielding a total level of functioning scale ranging from 0 to 7.

Classification of young adult outcomes

Each of these aspects of psychiatric disorder was tested as a predictor of two important young adult outcomes: secondary school completion and criminal involvement. These outcomes were assessed during the 1985–86 interviews.

Secondary school completion

Secondary school completion was determined by the young adult's report of whether s/he had completed the 12th grade or was on target to complete the 12th grade (no older than 18 years of age and in the 12th grade). The classification of school completion was vulnerable to error insofar as some of the 18 year olds who had not completed the 11th or 12th grade by the time of the wave 3 interview may eventually have graduated, and some of the young adults who were in the 12th grade at the time of the wave 3 interview may not have graduated. Data available from the CICS wave 4 interview in 1990 indicated that both of these types of misclassification were minimal.

Criminal involvement

Criminal involvement was determined by responses to questions directed to the young adult and the parent as to whether the young adult had “been in trouble with the police” within the two years before the 1985–86 interview. In addition, the structured psychiatric interview included questions about engagement in specific types of criminal activities, such as assaults, stealing, and property damage. The validity of the criminal involvement measure was assessed by comparing youth and parent reports of the youth being in trouble with police over the two years before the 1985–86 interview. As criminal behaviour is socially undesirable and sanctioned, youth would be expected to know more than their parents about their own criminal activities. Thus, we expected that most parental reports of their child's general criminal behaviour would also be reported by youth, but many of the youth's reports of criminal behaviour would not be corroborated by parents. For 138 (79%) of the 175 pairs of respondents to this question, both the youth and the parent endorsed “no.” Twelve pairs (7%) endorsed “yes.” In 21 (12%) cases, youth said “yes,” and parent said “no,” and in three (2%) cases, parent said “yes,” and youth said “no.” For the purposes of this study, a youth was considered to have criminal involvement on the basis of a positive endorsement from either reporter.

Classification of social class

Social class was measured by a scale constructed on the basis of mother's education, father's education, family income, and father's occupation.³⁶

Statistical analysis

Relative risks of adverse young adult outcomes and 95% confidence intervals were calculated for any psychiatric diagnosis, depressive diagnosis, anxiety diagnosis, disruptive diagnosis, and substance abuse diagnosis. Relative risks were also calculated for the “most disordered” quartile of total symptom score, disruptive symptom score, and adaptive functioning score. The entire study sample was used for each of the calculations. For example, the reference group for the relative risk of failure to complete school for adolescents with disruptive diagnoses was adolescents without disruptive diagnoses (not adolescents without any psychiatric diagnosis). The Maentel-Haenszel weighted approach with Greenland/Robins confidence limits (Epi Info, version 6) was used to calculate SES adjusted relative risks of school non-completion and gender adjusted relative risks of criminal involvement.

Two analytic strategies were used to assess the ability of different crude and adjusted measures of psychiatric disorder to predict each of the young adult outcomes: secondary school completion and criminal involvement. For the first analytic approach, using knowledge of whether the adverse young adult outcome had actually occurred as the “true status,” the sensitivity, specificity, and positive predictive value of adolescent psychiatric diagnosis and multiple cut off points of symptom and adaptive functioning scores were calculated. These analyses were conducted for the study sample as a whole and within two social class strata to predict secondary school completion. They were conducted for the sample as a whole and within gender subgroups to predict criminal involvement. As SES has a strong independent relation to secondary school completion and gender has a strong independent relation to criminal involvement, independent of the adolescent's mental health status, positive predictive value would likely be higher in the subgroup where the outcome was more common.

The second strategy entailed the application of logistic regression analysis. Psychiatric disorder, determined on the basis of diagnosis, symptoms, or adaptive functioning, was added to logistic models after baseline models were fitted with age, gender, and social class covariates. The logit predicted probability of school non-completion and the logit predicted probability of criminal involvement were calculated for each individual subject by multiplying the estimated coefficients by the subject's coded or actual values for each of the four independent variables.

Six sets of receiver operating characteristic (ROC) curves were constructed so that the predictive potential of the various strategies could be compared visually.^51,52 The ROC analysis extends the evaluation of predictive validity beyond sensitivity and specificity. ROC curves provide an easily comprehensible format for assessing a range of achievable levels of sensitivity and specificity based on a range of cut off values. This information can be taken into account both in choosing among different screening measures and in choosing a threshold value for one screening measure.⁵²

In this study, three ROC curves were drawn for each of the two young adult outcomes and for each approach to the determination of adolescent psychiatric disorder. In the first, prediction of outcome was based on various cut off values for the crude measure of psychiatric disorder (except diagnosis, which is binary). The second was based on various cut off values of predicted probability of outcome from logistic models including only age, gender, and SES, and the third was based on various cut off values of predicted outcome probability from logistic models including age, gender, SES, and the measure of psychiatric disorder.

RESULTS

Exposure

During adolescence, 56% of the 181 study participants were classified as having a psychiatric disorder based on either diagnosis, being in the upper 40th centile of symptom count, or being in the lower 40th centile of adaptive functioning. Fifteen per cent of the adolescents were classified as having a disorder based on all three approaches. Figure 1 depicts the overlap of the three approaches to measuring adolescent psychiatric disorder in the community sample. Of those meeting study diagnostic criteria, 87% were in the lowest 40% of adaptive functioning and/or the highest 40% of symptom count. Of those without diagnosis, 73% were in the highest 60% of functioning and the lowest 60% of symptom count.

Outcome

Of the adolescents in this study sample, 22 (12.2%) failed to complete secondary school. Twenty four (13.3%) had criminal involvement during their early adulthood. Of those who were involved with the police, 70% committed assaults and/or stole or damaged property.

Predicting outcome on the basis of exposure

Results of the tests of the ability of different measures of psychiatric disorder to predict young adult outcomes are presented separately for failure to complete secondary school and criminal involvement.

Failure to complete secondary school

Table 1 shows that regardless of the approach used to classify adolescents as having a psychiatric disorder, adolescents with a disorder were significantly more likely to fail to complete secondary school than those who were classified as not having a disorder. The crude relative risks ranged in magnitude from 2.86 (95% CI 1.32 to 6.18) for anxiety disorder diagnosis to 9.21 (95% CI 2.20 to 37.84) for the lower half of the distribution of adaptive functioning. Despite the apparent range in the magnitude of the relative risks among various approaches to classifying psychiatric disorder, the confidence intervals all overlapped. The different classification approaches varied widely with regard to sensitivity, specificity, and predictive value of a positive “test.”⁵³ By these measures of validity, being in the highest quartile on total number of psychiatric symptoms emerged as the most discriminative measure for identifying adolescents who did not complete secondary school, with a sensitivity of 0.73, a specificity of 0.82, and a PV+ of 0.36.

Table 2 shows the results of the subgroup analyses. Psychiatric disorder was strongly related to failure to complete school in both lower and upper SES strata. However, for each measure of psychiatric disorder, the PV+ for predicting school non-completion was somewhat higher in the lowest SES stratum. Using diagnosis, symptoms, or adaptive functioning for screening adolescents in the upper three SES quartiles yielded few true positives (12%–28% of youth who screened positive) relative to using these screening protocols for adolescents in the lowest quartile of SES. Because the occurrence of school non-completion was more common in this stratum, 46%–52% of those who screened positive were, in fact, true positives.

Three ROC curves were constructed to depict sensitivity and specificity for a range of cut off values of predicted probability of failing to complete school vis a vis true school completion status (fig 2). Predicted probability of the outcome was calculated on the basis of logistic models containing age, gender, and SES, and either (1) level of adaptive functioning score, (2) any diagnosis, or (3) number of symptoms. The ROC curves show that the logistic model with psychiatric symptoms yielded the highest specificity and sensitivity across most of the range of cut off values. For the psychiatric symptom model, the most efficient predicted probability cut off value was around p ≥ 0.13, where sensitivity was 0.89, and specificity was 0.87. Figure 2 shows that using combined knowledge of the adolescent's demographic characteristics and symptom status yielded an improvement in predictive ability compared with basing the prediction on either symptoms or demographic status alone. This was not true for the other two measures of disorder, where knowledge of the adolescent's age, gender, and social class, alone, yielded a better ability to predict future school completion than knowledge of adaptive functioning or diagnosis either alone or in addition to knowledge of demographic characteristics.

Criminal involvement

The relative risk of criminal involvement for adolescents with psychiatric disorder varied according to the classification approach used. Table 3 shows that the relative risks for the 25th and 40th centile cut off values of disruptive symptoms as well as for disruptive diagnosis were greater than 4.0. Both anxiety diagnosis and the 25th centile cut off for adaptive functioning yielded relative risks lower than 0.60. Because the “any diagnosis” and the “total psychiatric symptom” measures identified adolescents with both anxious and disruptive diagnoses or symptoms as positive, these two measures of disorder yielded intermediate relative risk estimate values of about 2. Only one of the approaches to measuring psychiatric disorder (highest 40th centile of disruptive symptoms) resulted in a sensitivity of over 0.60, and none resulted in a PV+ of higher than 0.35. The two measures with the best ability to predict future criminal involvement were disruptive symptoms and disruptive diagnosis.

Stratifying by gender revealed that all three aspects of adolescent psychiatric disorder were more accurate in predicting adult criminal involvement in boys than in girls (table 4). For boys, at 40th centile cut off value on the disruptive symptom scale, sensitivity was 0.80, specificity was 0.73, and PV+ was 0.47. Thus, in screening a community sample of boys using this approach to the classification of psychiatric disorder, 20% of the total sample were false positives, and 5% were false negatives. In girls, for whom criminal involvement was less common, the relatively large number of false positives overwhelmed the relatively small number of true positives regardless of the classification procedure used.

Figure 2 shows that across all values of predicted probability of criminal involvement, logistic models with adaptive functioning, disruptive diagnosis, and disruptive symptoms yielded only moderate sensitivity and specificity. The best discrimination of true outcome was made at the 0.15 predicted probability cut off with the model containing disruptive symptoms, where sensitivity was 0.75, and specificity was 0.76. The logistic model based on knowledge of the adolescent's disruptive symptoms and demographic characteristics demonstrated improved ability to predict young adult outcome status compared with knowledge of either disruptive symptoms or demographic status alone.

In summary, among the three aspects of adolescent psychiatric disorder examined, the number of psychiatric symptoms and the level of adaptive functioning were at least as informative as psychiatric diagnosis in predicting later school non-completion, particularly in the lowest SES quartile. The number of disruptive symptoms was at least as informative as disruptive disorder diagnosis as a predictor of young adult criminal involvement, particularly among boys.

DISCUSSION

Often the most complicated or expensive means of identifying persons at high risk of incurring a bad health outcome is also the most accurate, while the easiest and least expensive means—one that is more feasible to administer within large populations—is the least accurate. Assigning a childhood psychiatric diagnosis requires lengthy interviewing of multiple informants followed by the application of complex algorithms to multiple sets of responses. The question this study investigated was whether alternative ways of measuring psychiatric disorder, such as administering questionnaires in which symptoms or strengths can be counted and cut points applied, are as accurate as diagnosis in predicting young adult outcomes. Such protocols could be inexpensively implemented in school and community settings without the need for clinically trained staff.^54,55 The YAICS examined the relative validity of various measurement approaches applied during adolescence for predicting future failure to complete secondary school and involvement in criminal activities.

Using the “gold standard” of actual young adult outcome status, the sensitivity, specificity, and positive predictive value of having any psychiatric diagnosis was compared to a number of alternatives including:

(1) whether a child met diagnostic criteria for specific types of disorders,

(2) various cut off values for a scale reflecting the total number of psychiatric symptoms or disruptive symptoms endorsed, and

(3) various cut off values for a scale reflecting level of adaptive functioning.

The YAICS found that psychiatric diagnosis was not a superior predictor of later school failure or criminal involvement. The study showed that symptom counts or assessment of level of functioning predicted just as well as, if not better than, diagnosis. Both of these measurement approaches would be easier to use in community-based screening protocols. This is not to say that a psychiatric diagnosis has no utility in guiding clinical decisions or in measuring the prevalence of psychiatric disorders. However, in this study, psychiatric diagnosis was not shown to yield a more accurate prediction of future outcomes than the simpler measures of disorder.

Most longitudinal studies of adolescents with psychiatric disorders have focused attention on a particular diagnosis, such as depression or conduct disorder.^56–62 Pooling the findings from prior studies, one could make the inference that in general, adolescents with psychiatric disorder fare poorly. One of the unique contributions of this study is that, as a full diagnostic interview was applied to a community-based sample, equivalent methods were used to document the life course of youth with a variety of different mental health problems. Thus we have seen that adolescents with disruptive and substance abuse disorder diagnoses are at high risk of failing to complete school, as are adolescents with depressive and anxiety disorder diagnoses. For the outcome of criminal involvement, a different picture emerges. Whereas adolescents with disruptive, substance abuse, and depressive disorder diagnoses are at increased risk of future criminal involvement, having an anxiety disorder diagnosis is associated with decreased risk. This finding is consistent with the results of a treatment-based follow up study, which showed that the typical young adult outcomes of adolescents with psychiatric disorders varies by diagnostic group.¹³ Such findings warn against drawing broad conclusions about the impact of all varieties of adolescent psychiatric disorder on all young adult outcomes.

A methodological point is illustrated in the subgroup analyses where the relative risks were quite consistent across subgroups, but the positive predictive values varied widely. In subgroups where the outcome was relatively uncommon (for example, lack of school completion in upper SES strata, criminal involvement in girls), regardless of the total proportion of “screen positives,” the number of false positives is much higher than the number of true positives. The implication of this phenomenon for developing screening protocols is that scarce resources may be “misspent” if individuals in subgroups with a low prevalence of the adverse outcome are referred to prevention programmes simply on the basis of screening positive.

Limitations

Small sample size limits our ability to draw firm conclusions about the validity of different methods of classifying psychiatric disorder within population subgroups. With a larger study, the confidence intervals around estimates of the magnitude of effects of various measures of psychiatric disorder on young adult outcomes would be narrower. It would be possible to test for statistically significant differences in the validity of specific measures of disorder for predicting young adult outcomes, as well as for significant differences in the validity of specific measures applied within population subgroups. As it stands, the consistency of study findings when different analytic approaches are used strengthens the ability to draw tentative conclusions. Perhaps the greatest contribution of this study lies in demonstrating the use of a variety of methodological approaches for tackling a question that has important public health implications.

A second limitation has to do with the generalisability of the study findings. Although the methodological approaches are useful, the actual results must be interpreted in light of the fact that, since the 1980s, major revisions have been made to the psychiatric nomenclature, to structured psychiatric diagnostic instruments, and to measures of adaptive functioning. For example, the validity of the unidimensional Child Global Assessment Scale ⁴⁵ and the multidimensional Columbia Impairment Scale ⁶³ and the Child and Adolescent Functional Assessment Scale ⁶⁴ have been established. With improved ability to measure adaptive functioning, this aspect of psychiatric disorder may have stronger predictive ability than has been demonstrated in this study.

Implications of study findings

For many years, researchers and clinicians have struggled to develop operational definitions of childhood psychiatric disorders that have practical value in identifying those who need treatment and in guiding decisions regarding appropriate intervention. In recent years, in the United States the designation of Serious Emotional Disturbance (SED) has moved to the forefront for defining children whose psychiatric disorder is causing disability and who should be given highest priority for limited service resources. The operational definition of SED varies across service systems and geographical locales, but typically takes into account the presence of a diagnosable psychiatric disorder as well as impairment in the child's ability to function in developmentally normative activities within home, school, and community.⁶⁵

This study demonstrated the application of several methodological approaches to answering the question of how to identify youth who are at high risk of future trouble. Counting psychiatric symptoms emerged as a promising approach for predicting young adult outcomes. The best screening protocol for current usage could be determined by subjecting the more recently developed functional assessment scales, symptom checklists, and diagnostic interview schedules to the analytic methods used in this study. Future studies could also clarify the risks and benefits of applying one size fits all screening criteria versus applying different screening measures or cut off values depending on the gender or socioeconomic circumstances of the child.