How generalizable are coronary risk prediction models? Comparison of Framingham and two national cohorts

doi:10.1016/S0002-8703(99)70407-2

American Heart Journal

Volume 137, Issue 5, May 1999, Pages 837-845

https://doi.org/10.1016/S0002-8703(99)70407-2 Get rights and content

Abstract

Background Previous models used to predict individual risk of death from coronary heart disease (CHD) were developed from data of 3 decades ago from the Framingham Heart Study. CHD mortality rates have declined markedly since that period as a result of improvement in both risk factor status and medical interventions. Generalization of the results from this one study to the population at large remains a matter of concern. We compared predictive functions derived from the major risk factors for CHD from Framingham and 2 more recent national cohorts, the First and Second National Health and Nutrition Examination Survey (NHANES I and NHANES II). Methods and Results The participants included 1846 men and 2323 women 35 to 69 years of age and free of CHD at the fourth examination (1954 to 1958) from the Framingham Study; 2753 men and 3858 women from the NHANES I (1971 to 1975); and 2655 men and 3050 women from NHANES II (1976 to 1980). The 3 cohorts were monitored for 24, 20, and 15 years, respectively. Significant heterogeneity existed among studies in the magnitude of the Cox coefficients for the individual factors (ie, age, systolic blood pressure, serum total cholesterol, and smoking status), especially among men. When risk factors were considered collectively, however, functions derived from and applied to different cohorts had a similar ability to rank individual risk. The areas under the receiver operating characteristic curves were 0.71 to 0.76 in men and 0.76 to 0.81 in women when different risk functions were applied to their own population or to a second population. The cumulative CHD survival observed in women in the 2 national cohorts was close to what was predicted from the Framingham equation. However, Framingham overestimated the cumulative CHD mortality rates in men in NHANES I and NHANES II. Conclusions The Framingham risk model for the prediction of CHD mortality rates provides a reasonable rank ordering of risk for individuals in the US white population for the period 1975 to 1990. However, prediction of absolute risk is less accurate. (Am Heart J 1999;137:837-45.)

Section snippets

The Framingham Heart Study

The study was initiated in 1948 to investigate the factors associated with the development of cardiovascular disease in a representative sample of the adult population of Framingham, Massachusetts.¹³ A total of 5209 men and women 29 to 62 years of age were examined during the 4-year period of 1948 to 1952 and subsequently recalled for examinations every 2 years. The fourth examination, which took place during 1954 to 1958, was chosen as the baseline for the current report because the serum

Results

The baseline characteristics, number of CHD deaths, and age-adjusted mortality rates for the three cohorts are presented in Table I.

. Baseline characteristics, number of deaths, and age-adjusted mortality rates in follow-up: Framingham Heart Study, NHANES I, and NHANES II Studies

Empty Cell	Mean	SD	Mean	SD	Mean	SD
Empty Cell	Framingham (1954-1958)		NHANES I (1971-1975)		NHANES II (1976-1980)
Men
No. at baseline	1846		2753		2655
Age (y)	49.6	8.5	53.2	10.5	54.3	10.5
Systolic blood pressure (mm Hg)	132.0	20.0	135.6	19.7	133.4	19.3
	(133.7)	(135.3)	(132.6)

Discussion

The cohorts that were examined here, although sampled with entirely different strategies in different historical eras, provide evidence that the major risk factors are associated with CHD mortality rates among different populations and among men and women. The notable exception to this conclusion is smoking, in which heterogeneity was substantial. Despite heterogeneity in individual factors, when the risk factors were considered collectively, the ability of different models to rank individuals

References (29)

WB Kannel et al.
A general cardiovascular risk profile: the Framingham Study
Am J Cardiol
(1976)
PE Leaverton et al.
Representativeness of the Framingham risk model for coronary heart disease mortality: a comparison with a national cohort study
J Chron Dis
(1987)
A Menotti et al.
Twenty-five-year mortality from coronary heart disease and its prediction in five cohorts of middle-aged men in Finland, The Netherlands, and Italy
Prev Med
(1990)
T Gordon et al.
Predictability of coronary heart disease
J Chron Dis
(1979)
RJ Brand et al.
Multivariate prediction of coronary heart disease in the Western Collaborative Group Study compared to the findings of the Framingham Study
Circulation
(1976)
LE Chambless et al.
On the use of a logistic risk score in predicting risk of coronary heart disease
Stat Med
(1990)
ERICA Research Group
Prediction of coronary heart disease in Europe: the 2nd report of the WHO-ERICA Project
Eur Heart J
(1991)
A Menotti et al.
The prediction of coronary heart disease mortality as a function of major risk factors in over 30,000 men in the Italian RIFLE Pooling Project: a comparison with the MRFIT primary screenees
J Cardiovasc Risk
(1994)
A Menotti et al.
Coronary mortality and its prediction in samples of US and Italian railroad employees in 25 years within the Seven Countries Study of Cardiovascular Diseases
Int J Epidemiol
(1995)
D McGee et al.
The results of the Framingham Study applied to 4 other US based epidemiologic studies of cardiovascular disease

Pooling Project Research Group

Relationship of blood pressure, serum cholesterol, smoking habit, relative weight, and ECG abnormalities to incidence of major coronary events: final report of the Pooling Project

J Chron Dis

(1978)

C Sempos et al.

Divergence of the recent trends in coronary mortality for the 4 major race-sex groups in the United States

Am J Public Health

(1988)

TR Dawber et al.

Epidemiological approaches to heart disease: the Framingham Study

Am J Public Health

(1951)

National Center for Health Statistics

Plan and Operation of the Health and Nutrition Examination Survey, United States, 1971-73

(1979)

Cited by (104)

Traditional Cardiovascular Risk Factors Strongly Underestimate the 5-Year Occurrence of Cardiovascular Morbidity and Mortality in Spinal Cord Injured Individuals
2021, Archives of Physical Medicine and Rehabilitation
To explore whether traditional models of cardiovascular disease (CVD) risk prediction correctly predict CVD events across a median 5.7-year follow-up period in individuals with spinal cord injury (SCI) and whether adding SCI-related characteristics (ie, lesion level) to the prediction model can improve the prognostic value.
Retrospective analysis of patient records.
Observation at the start of active rehabilitation of participants in a multicenter cohort study, “Restoration of (Wheelchair) Mobility in SCI Rehabilitation,” in the Netherlands.
Patients with SCI (N=200) The patients were 74% men, aged 40±14 years, and with an American Spinal Injury Association (ASIA) impairment score of A through D. Forty percent had tetraplegia, and 69% were motor complete.
Risk profiling/not applicable.
Survival status and cardiovascular morbidity and mortality qwere obtained from medical records. Five-year Framingham Risk Scores (FRS) and the FRS ability to predict events assessed using receiver operating characteristic (ROC) curves with corresponding areas under the curve (AUC) and 95% confidence intervals (CI). Kaplan-Meier curves and the log-rank test were used to assess the difference in clinical outcome between participants with an FRS score lower or higher than the median FRS score for the cohort. SCI-related factors associated with CVD events, ASIA impairment, motor completeness, level of injury, and sports participation before injury were explored using univariate and multivariate Cox proportional hazard regression.
The median 5-year FRS was 1.36%. Across a median follow-up period of 5.7 years, 39 developed a CVD event, including 10 fatalities. Although the FRS markedly underestimated the true occurrence of CVD events, the Kaplan-Meier curves and the log-rank test showed that the risk ratio for individuals with an FRS score less than the median FRS (eg, low risk) versus a score greater than the median FRS (high risk) was 3.2 (95% CI, 1.6-6.5; P=.001). Moreover, ROC with corresponding AUCs suggests acceptable accuracy of the FRS to identify individuals with increased risk for future CVD events (ROC AUC of 0.71; 95% CI, 0.62-0.82). Adding ASIA impairment (0.74; 95% CI, 0.66-0.82), motor impairment (0.74; 95% CI, 0.66-0.83), level of injury (0.72; 95% CI, 0.63-0.81), or active engagement in sport before injury (0.72; 95% CI, 0.63-0.88) to the FRS did not improve the level of discrimination.
Our 5.7-year retrospective study reveals that cardiovascular risk factors and risk models markedly underestimate the true risk for CVD events in individuals with SCI. Nonetheless, these markers successfully distinguish between SCI individuals at high versus low risk for future CVD events. Our data may have future clinical implications, both related to (cutoff values of) CVD risk factors, but also for (earlier) prescription of (non)pharmacologic strategies against CVD in SCI individuals.
Traumatic transfers: Calibration is adversely affected when prediction models are transferred between trauma care contexts in India and the United States
2016, Journal of Clinical Epidemiology
We evaluated the transferability of prediction models between trauma care contexts in India and the United States and explored updating methods to adjust such models for new contexts.
Using a combination of prospective cohort and registry data from 3,728 patients of Towards Improved Trauma Care Outcomes in India (TITCO) and from 18,756 patients of the US National Trauma Data Bank (NTDB), we derived models in one context and validated them in the other, assessing them for discrimination and calibration using systolic blood pressure, heart rate, and Glasgow coma scale as candidate predictors.
Early mortality was 8% in the TITCO and 1–2% in the NTDB samples. Both models discriminated well, but the TITCO model overestimated the risk of mortality in NTDB patients, and the NTDB model underestimated the risk in TITCO patients.
Transferability was good in terms of discrimination but poor in terms of calibration. It was possible to improve this miscalibration by updating the models' intercept. This updating method could be used in samples with as few as 25 events.
Child abuse/neglect risk assessment under field practice conditions: Tests of external and temporal validity and comparison with heart disease prediction
2015, Children and Youth Services Review
(1) Identify validation design and accuracy assessment standards for medical prognostic models applicable to evaluation of child abuse/neglect (CA/N) risk assessment models. (2) Assess the accuracy of the California Family Risk Assessment (CFRA) in predicting CA/N using the foregoing standards. (3) Compare the prediction accuracy of the CFRA with the prediction accuracy of coronary heart disease (CHD) prediction models.
(1) What validation design and accuracy assessment standards are used to evaluate medical prognostic models? (2) What is the evidence for the accuracy of the CFRA using those standards? (3) How does the accuracy of the CFRA in predicting CA/N compare with the accuracy of CHD prediction models, which are a reasonable exemplar for the CA/N prediction effort?
An external validation sample of 236 California reports of CA/N from San Luis Obispo and Sutter counties, and a larger temporal validation sample of 6307 California reports from Orange, Los Angeles, and Humboldt counties were investigated and assessed with the CFRA by line child welfare staff and were followed prospectively statewide for two years to discover reported, substantiated CA/N in any California county. CFRA accuracy in predicting substantiated CA/N was assessed by calibration and discrimination. Calibration was measured as the ratio of predicted to observed cases of CA/N seen during follow-up, with a ratio of 1.0 registering perfect calibration. Discrimination was measured by the area under the receiver operating characteristic (ROC) curve (AUC), with values from .60 to .85 found typical for medical prognostic models. CHD prediction literature was reviewed to acquire values of these accuracy measures for CHD prediction models. CFRA CA/N prediction accuracy and CHD prediction accuracy were then compared.
Findings from external and temporal validation samples support the accuracy of CFRA prediction of CA/N. CFRA accuracy in predicting CA/N compared well with CHD prediction accuracy: (1) in the external validation sample, 43.42 CA/N cases were predicted during follow-up and 47 were observed, with consequent 7.6% deviation from perfect calibration. (2) In the temporal validation sample 857.49 CA/N cases were predicted and 801 were observed, with 7.1% deviation from perfect calibration. (3) The best performing of 20 Framingham CHD prediction models identified by systematic literature review predicted 222 CHD cases and 206 were observed, with 7.8% deviation from perfect calibration. (3) The CFRA external and temporal validation sample AUCs were .74 and .64, respectively. (4) For 26 CHD prediction cohorts found by literature review, the AUC mean and median values were .72 and .71, respectively, with a range from .60 to .84.
(1) External and temporal validation results support the accuracy of the CFRA. (2) CFRA CA/N prediction accuracy parity with that for CHD prediction is encouraging, suggesting that wide use of the CFRA, properly implemented, could improve risk assessment accuracy in child protection. (3) Findings underline the importance of ensuring that no risk assessment model or method, including actuarial and consensus models and clinical judgment, is used in the field unless it has passed a test of external, or at least temporal validation.
Family history and body mass index predict perceived risks of diabetes and heart attack among community-dwelling Caucasian, Filipino, Korean, and Latino Americans-DiLH Survey
2015, Diabetes Research and Clinical Practice
The purpose of the study was to explore the perceived risk for diabetes and heart attack and associated health status of Caucasian, Filipino, Korean, and Latino Americans without diabetes.
A cross-sectional survey was conducted with 904 urban adults (mean age 44.3 ± 16.1 years; 64.3% female) in English, Spanish or Korean between August and December 2013.
Perceived risk for developing diabetes was indicated by 46.5% (n = 421), and 14.3% (n = 129) perceived themselves to be at risk for having a heart attack in their lifetime. Significant predictors of pessimistic diabetes risk perceptions: Filipino (adjusted odds ratio [AOR] = 1.7; 95% CI: 1.04–2.86) and Korean (AOR = 2.4; 1.33–4.48) ethnicity, family history of diabetes (AOR = 1.4; 1.00–1.84), female gender (AOR = 1.4; 1.04–1.96), high cholesterol (AOR= 1.6; 1.09–2.37) and higher body mass index (BMI) (AOR = 1.1; 1.08–1.15). Predictors of pessimistic heart attack risk perceptions were family history of an early heart attack (AOR = 2.9; 1.69–5.02), high blood pressure (AOR = 2.4; 1.45–3.84), and higher BMI (AOR = 1.1; 1.04–1.12) after controlling for socio-demographic factors. Older age, physical inactivity, smoking, and low HDL levels were not associated with risk perceptions.
Multiple risk factors were predictive of greater perceived diabetes risk, whereas, only family history of heart attack, high blood pressure and increases in BMI significantly contributed to perceived risk of heart attack among ethnically diverse at risk middle-aged adults. It is important that healthcare providers address the discordance between an individual's risk perceptions and the presence of actual risk factors.
The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions
2014, Journal of Biomedical Informatics
Hospital readmission risk prediction remains a motivated area of investigation and operations in light of the hospital readmissions reduction program through CMS. Multiple models of risk have been reported with variable discriminatory performances, and it remains unclear how design factors affect performance.
To study the effects of varying three factors of model development in the prediction of risk based on health record data: (1) reason for readmission (primary readmission diagnosis); (2) available data and data types (e.g. visit history, laboratory results, etc); (3) cohort selection.
Regularized regression (LASSO) to generate predictions of readmissions risk using prevalence sampling. Support Vector Machine (SVM) used for comparison in cohort selection testing. Calibration by model refitting to outcome prevalence.
Predicting readmission risk across multiple reasons for readmission resulted in ROC areas ranging from 0.92 for readmission for congestive heart failure to 0.71 for syncope and 0.68 for all-cause readmission. Visit history and laboratory tests contributed the most predictive value; contributions varied by readmission diagnosis. Cohort definition affected performance for both parametric and nonparametric algorithms. Compared to all patients, limiting the cohort to patients whose index admission and readmission diagnoses matched resulted in a decrease in average ROC from 0.78 to 0.55 (difference in ROC 0.23, p value 0.01). Calibration plots demonstrate good calibration with low mean squared error.
Targeting reason for readmission in risk prediction impacted discriminatory performance. In general, laboratory data and visit history data contributed the most to prediction; data source contributions varied by reason for readmission. Cohort selection had a large impact on model performance, and these results demonstrate the difficulty of comparing results across different studies of predictive risk modeling.
2013 ACC/AHA guideline on the assessment of cardiovascular risk: A report of the American college of cardiology/American heart association task force on practice guidelines
2014, Journal of the American College of Cardiology

View all citing articles on Scopus

^☆: Supported by a grant from the National Heart, Lung, and Blood Institute (HL-58436).

^☆☆: Reprint requests: Youlian Liao, MD, Department of Preventive Medicine and Epidemiology, Loyola University Stritch School of Medicine, 2160 S First Ave, Maywood, IL 60153.

^★: 0002-8703/99/$8.00 + 0 4/1/93037

View full text

How generalizable are coronary risk prediction models? Comparison of Framingham and two national cohorts☆,☆☆,★

Abstract

Section snippets

The Framingham Heart Study

Results

Discussion

Am J Cardiol

J Chron Dis

Prev Med

J Chron Dis

Multivariate prediction of coronary heart disease in the Western Collaborative Group Study compared to the findings of the Framingham Study

Circulation

On the use of a logistic risk score in predicting risk of coronary heart disease

Stat Med

Prediction of coronary heart disease in Europe: the 2nd report of the WHO-ERICA Project

Eur Heart J

The prediction of coronary heart disease mortality as a function of major risk factors in over 30,000 men in the Italian RIFLE Pooling Project: a comparison with the MRFIT primary screenees

J Cardiovasc Risk

Coronary mortality and its prediction in samples of US and Italian railroad employees in 25 years within the Seven Countries Study of Cardiovascular Diseases

Int J Epidemiol

The results of the Framingham Study applied to 4 other US based epidemiologic studies of cardiovascular disease

Relationship of blood pressure, serum cholesterol, smoking habit, relative weight, and ECG abnormalities to incidence of major coronary events: final report of the Pooling Project

J Chron Dis

Divergence of the recent trends in coronary mortality for the 4 major race-sex groups in the United States

Am J Public Health

Epidemiological approaches to heart disease: the Framingham Study

Am J Public Health

Plan and Operation of the Health and Nutrition Examination Survey, United States, 1971-73