Article Text
PDF
Abstract
Background The Veterans Health Administration COVID-19 (VACO) Index predicts 30-day all-cause mortality in patients with COVID-19 using age, sex and pre-existing comorbidity diagnoses. The VACO Index was initially developed and validated in a nationwide cohort of US veterans—we now assess its accuracy in an academic medical centre and a nationwide US Medicare cohort.
Methods With measures and weights previously derived and validated in US national Veterans Health Administration (VA) inpatients and outpatients (n=13 323), we evaluated the accuracy of the VACO Index for estimating 30-day all-cause mortality using area under the receiver operating characteristic curve (AUC) and calibration plots of predicted versus observed mortality in inpatients at a single US academic medical centre (n=1307) and in Medicare inpatients and outpatients aged 65+ (n=427 224).
Results 30-day mortality varied by data source: VA 8.5%, academic medical centre 17.5%, Medicare 16.0%. The VACO Index demonstrated similar discrimination in VA (AUC=0.82) and academic medical centre inpatient population (AUC=0.80), and when restricted to patients aged 65+ in VA (AUC=0.69) and Medicare inpatient and outpatient data (AUC=0.67). The Index modestly overestimated risk in VA and Medicare data and underestimated risk in Yale New Haven Hospital data.
Conclusions The VACO Index estimates risk of short-term mortality across a wide variety of patients with COVID-19 using data available prior to or at the time of diagnosis. The VACO Index could help inform primary and booster vaccination prioritisation, and indicate who among outpatients testing positive for SARS-CoV-2 should receive greater clinical attention or scarce treatments.
- epidemiology
- mortality
- COVID-19
- public health policy
Data availability statement
Data may be obtained from a third party and are not publicly available. The United States Department of Veterans Affairs (VA) places legal restrictions on access to veteran’s health care data, which includes both identifying data and sensitive patient information. The analytic data sets used for this study are not permitted to leave the VA firewall without a Data Use Agreement. This limitation is consistent with other studies based on VA data. However, VA data are made freely available to researchers behind the VA firewall with an approved VA study protocol. For more information, please visit https://www.virec.research.va.gov or contact the VA Information Resource Center (VIReC) at VIReC{at}va.gov. CareJourney has an existing Data Use Agreement with CMS that prohibits disclosure of patient health care data. The datasets used for this study are not permitted to be accessed unless users have an existing Data Use Agreement with CMS for the necessary datasets. This limitation is consistent with studies done by entities with access to Medicare Claims through the CMS Innovators' License.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Current age and pre-existing medical condition criteria for COVID-19 vaccine prioritisation1 2 identify a significant proportion of the world adult population, exceeding available vaccine supplies. A more tailored risk estimation combining age with pre-existing conditions could provide a more granular approach to prioritisation,3 and play a role in: (1) motivating high-risk individuals and their contacts to practise social distancing until vaccinated, (2) identifying individuals testing positive at drive-up sites who require clinical examination and possibly laboratory evaluation, and (3) prioritising patients for primary or booster vaccination or scarce treatments.
Hundreds of publications have identified risk factors associated with adverse COVID-19 outcomes,4 5 but just a small subset have developed predictive models using electronic health record (EHR) or administrative data available prior to infection and that are readily analysable, enabling rapid and large-scale identification of those at greatest risk should they become infected.6–10 Even fewer studies have prospectively validated their models in large samples. The Veterans Health Administration COVID-19 (VACO) Index for short-term, all-cause mortality is based on age, sex and comorbid diagnoses, and was developed and validated using EHR data from a nationwide cohort from the US Veterans Health Administration (VA).11 Within VA, the VACO Index demonstrated good discrimination across time intervals and among men and women, White, Black and Hispanic patients and those living in different geographic regions. The Index is available as a web application at MDCalc.com and as a mobile application for iOS and Android platforms. An EHR decision support tool which autopopulates Index data elements is currently in development.
Some have questioned the generalisability of VA data for understanding risk of mortality from COVID-19. US veterans in care are predominantly male, older and have a higher prevalence of chronic health conditions and risk behaviours than the US population.12–14 To address these concerns, we compare the accuracy (discrimination and calibration) of VACO Index short-term mortality estimates for patients with COVID-19 in: (1) our original US nationwide VA sample of 13 323 inpatients and outpatients, (2) 1307 patients admitted to Yale New Haven Hospital (YNHH, a tertiary care academic medical centre drawing from Southern New England, New York state and beyond), and (3) 441 854 US nationwide Medicare recipients (aged 65 and older). We consider the accuracy of the Index overall and within important subgroups defined by calendar time, age, sex, race/ethnicity and geographic region. We also consider whether different weighting of the index components or adding race/ethnicity or body mass index (BMI) improves the accuracy of the VACO Index.
Methods
VACO Index components
The VACO Index, a 30-day all-cause mortality prediction model developed and validated in VA nationwide data, uses demographic and pre-existing condition data available in EHR or medical administrative data.11 The Index includes age, sex, multimorbidity quantified with the Charlson Comorbidity Index (CCI) derived from International Classification of Diseases, 10th Edition (ICD-10) diagnosis codes (online supplemental file 1)15 16 and myocardial infarction (MI) or peripheral vascular disease (PVD)—neither race nor any other individual comorbid diagnosis provided additional discriminatory function (online supplemental file 2). The VACO Index had an area under the receiver operating characteristic curve (AUC) of 0.79 in development, 0.81 in early validation and 0.84 in later validation, and validated well in sex, race/ethnicity and regional subgroups.
Supplemental material
Supplemental material
Data source and participants
VA data
We used a nationwide sample of 13 323 veterans testing positive for SARS-CoV-2 in the inpatient or outpatient setting between 2 March 2020 and 15 April 2020 and followed for 30 days. VA data were split into development (test positive 2 March 2020 to 15 April 2020), early validation (16 April 2020 to 18 May 2020) and later validation (19 May 2020 to 19 July 2020) cohorts. Deaths were determined using inpatient records and the VA death registry to capture deaths occurring outside hospitalisation.
YNHH data
We extracted data from the YNHH Epic (Epic Systems, Madison, Wisconsin, USA) EHR on 1307 patients testing positive for SARS-CoV-2 between 2 March 2020 and 19 July 2020 who were admitted to the hospital. Patients were eligible for inclusion if they tested positive within 14 days before an admission, on the day of admission or while hospitalised. Similar to the VACO Index methodology, we used the first SARS-CoV-2 positive test date, and baseline was the date of positive testing for those admitted within 14 days after testing positive, or date of testing for those testing positive more than 14 days after admission (presumably a nosocomial infection).
Thirty-day deaths were determined by index hospitalisation discharge status and postdischarge deaths captured by the healthcare system EHR. YNHH data were restricted to inpatient deaths during the YNHH index and subsequent admissions, and any outpatient deaths recorded in the YNHH EHR, so we did not have complete 30-day mortality capture as in the VA data.
Medicare data
We used US Medicare Fee for Service data to identify 427 224 patients nationwide aged 65 or older with an inpatient or outpatient COVID-19 diagnosis between 2 March 2020 and 19 July 2020, using ICD-10 diagnosis code of B97.29 on pre-April data and U07.1 stating 1 April 2020. Deaths were ascertained by discharge status after any Medicare hospitalisation, supplemented with deaths recorded in the Medicare database.
Data harmonisation
For all data sets, we categorised data into identical age, race, and sex strata and determined CCI scores from ICD-10 diagnosis codes. If an individual had a COVID-19 diagnosis code on more than one date, we only included the first date. Data were split into ‘early’ and ‘later’ cohorts temporally synchronised with the VACO Index development cohort and combined validation early and late cohorts. We used an identical 30-day follow-up window for determining death. We explored the more granular race stratification available in Medicare data (non-Hispanic White, non-Hispanic Black, Hispanic, Asian, North American Native, Other). In VA and Medicare data sets we also considered the most recent BMI recorded between 730 and 15 days before baseline.
Statistical analyses
We used multivariable logistic regressions to model 30-day all-cause mortality. To account for missing BMI data in the VA and YNHH data sets, we used multiple imputation by chained equations, including all predictor and outcome variables and auxiliary variables, generated 10 imputed data sets for the development and validation cohorts, and proportions, ORs and 95% CIs were combined according to Rubin’s rules.17
We report the AUC and calibration plots as assessments of the VACO Index performance and of alternative models in all three data sets. We used the VACO Index coefficients to calculate a predicted mortality for each patient in the YNHH and Medicare data sets, and then determined AUCs for the early and later testing dates, and in important subgroups: age (<65 vs 65+, except in Medicare data), sex (male vs female), race/ethnicity (Black vs non-Black), US census region (except in YNHH data), and compared the values to those obtained in the original VACO Index development cohort. We refit models using the VACO Index variables in the VA, YNHH and Medicare data sets to understand how the model might have differed had it been developed in alternative data sets. To assess the impact of race and BMI, we also refit models supplementing VACO Index variables with race (VA, YNHH and Medicare) and BMI (VA, YNHH).
We assessed calibration plots of observed versus predicted 30-day mortality in 10 (VA and Medicare data) or five (small sample size: YNHH data, VA female subgroup) strata containing equal numbers of deaths in early and later testing cohorts and in subgroups by age, sex, race/ethnicity and US census region. Data analyses were performed using Stata V.15.1 (StataCorp, College Station, Texas, USA) and SAS V.9.4 (SAS Institute). This study was conducted in compliance with the Health Insurance Portability and Accountability Act. Neither the patients nor the public were involved in the design, conduct, reporting or dissemination of this research. Dissemination of results to deidentified subjects is not practicable. This cohort study is reported according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guidelines (online supplemental file 3).18
Supplemental material
Results
Participants
The data set included 13 323 VA, 1307 YNHH and 427 224 Medicare patients for a total of 441 854 patients with COVID-19 (online supplemental file 4). Compared with VA (median age 63 years) and YNHH (65 years), Medicare patients (78 years) were older; a higher proportion of Medicare patients were White (75%) compared with YNHH (44%) and VA (39%); YNHH had the largest proportion of Hispanic patients (21%). Women were better represented in YNHH (52%) and Medicare (58%) than in VA (9%). Consistent with the older age of the Medicare cohort, most comorbid conditions were substantially more common.
Supplemental material
Performance in the YNHH cohort
The VACO Index demonstrated good discrimination in YNHH data overall (AUC: 0.80, 95% CI 0.77 to 0.83), consistent with that seen in VA overall (AUC: 0.82, 95% CI 0.81 to 0.83) (table 1). Discrimination in YNHH early and later periods was similar to the VA development period. When AUCs were estimated for subgroups, the VACO Index demonstrated similar discrimination in YNHH data as in VA data among those <65 years of age, those 65 and over, men, non-Hispanic Black patients, Hispanic patients and Other race/ethnicity patients. The discrimination among non-Hispanic White patients at YNHH was not as strong (AUC: 0.72, 95% CI 0.67 to 0.77) as in VA (AUC: 0.82, 95% CI 0.81 to 0.84), while good discrimination among women in YNHH (AUC: 0.81, 95% CI 0.77 to 0.85) was not as strong as among women in VA (AUC: 0.88, 95% CI 0.81 to 0.95).
Validation of VACO Index 30-day COVID-19 mortality estimates in subgroups of VA, YNHH and Medicare cohorts
Performance in the Medicare cohort
Restricting VA data to the age range included in Medicare (65 years or older), the VACO Index demonstrated consistent AUCs in VA and Medicare data (table 2). In all cases, the AUCs estimated within Medicare data fell within the 95% CI for the VA AUCs overall and by testing date, sex, race/ethnicity and census region.
Impact of adding race and BMI to VACO Index 30-day COVID-19 mortality estimates in VA, YNHH and Medicare cohorts
The VACO Index modestly overestimated mortality in VA data and Medicare population, underestimated risk in the YNHH population (figure 1) and performed well in subgroups (online supplemental file 5). Of note, the Index slightly underestimated risk in the early interval of Medicare data but was better calibrated among Black, Hispanic, Asian and North American Native than among White patients.
Supplemental material
Calibration plots comparing Veterans Health Administration COVID-19 (VACO) Index predicted and observed 30-day mortality in (A) Veterans Health Administration (VA) data, (B) Yale New Haven Health (YNHH), and (C) Medicare data. The dashed diagonal line represents perfect prediction. Values above the dashed line indicate overprediction of mortality by the VACO Index, and values below the line represent underprediction. Error bars depict 95% CIs of mortality predictions.
VACO Index supplemented with race and BMI
Compared with the original VACO Index (table 2, model 1), the addition of race and BMI did not improve AUCs in VA, YNHH or Medicare data (models 2 and 3). Refitting the VACO Index variables in YNHH and Medicare data improved the AUC by a single point (model 4), but adding race and BMI to the refit did not further improve the AUCs (models 5 and 6). Fully adjusted models AUCs (models 7–12) fell largely within the CIs estimated in the earlier models. Forest plots of ORs from the VACO Index and refitted models in VA, YNHH and Medicare data using VACO Index variables supplemented with race and BMI showed similar ORs for age, sex, race, CCI, MI or PVD, and BMI (figure 2).
Forest plot of multivariable models of COVID-19 30-day mortality. Veterans Health Administration COVID-19 (VACO) Index original model includes age, sex, Charlson Comorbidity Index and age interaction term, and myocardial infarction (MI) or peripheral vascular disease (PVD). Veterans Health Administration (VA) data model refits VACO Index variables and adds body mass index (BMI) and race (non-Hispanic White, non-Hispanic Black, Hispanic, Other). Yale New Haven Health (YNHH) data model refits VACO Index variables and adds BMI and race (non-Hispanic White, non-Hispanic Black, Hispanic, Other). Medicare data model refits VACO Index variables and adds additional race categories (non-Hispanic White, non-Hispanic Black, Hispanic, Asian North American Native, Other). Models and ORs of most variables are consistent across the data sets, providing evidence of the generalisability of the VACO Index model.
Discussion
Based on age, sex and pre-existing conditions, the VACO Index maintains good discrimination and calibration for short-term mortality among patients with COVID-19 in two demographically diverse samples outside the VA healthcare system. Adjusted associations identified in VA data are largely consistent in YNHH and Medicare data, demonstrating that relative weights assigned variables in the VACO Index are reasonably consistent with the weights that would be assigned with YNHH or Medicare data. These findings suggest that COVID-19 mortality risk factors seen in VA data generalise to other US populations. Additionally, even though mortality rates have decreased over time, the index effectively discriminates those at greater risk of mortality in both time periods within all three samples. This is true regardless of patient sex, race/ethnicity or region of the country.
The VACO Index demonstrates that while age is a major driver of COVID-19 mortality, comorbid disease burden quantified by CCI further stratifies risk. When we separately modelled all 17 conditions in the CCI individually, the AUCs were improved by 0.01. However, the weighting assigned to the individual conditions varied more across cohorts than did the weighting assigned the CCI score (data not otherwise shown)—thus a model fit to individual conditions would be less likely to generalise as well to new data. Furthermore, we found additional predictive value in including MI and PVD as individual variables, in addition to their contribution to the CCI score.
Given the ready accessibility of the VACO Index on MDCalc.com, we anticipate several immediate applications. An important issue is management of those undergoing testing in drive-up settings. A follow-up phone call to notify an individual that they are positive currently includes a few questions about fever and shortness of breath. If neither are present the patient is advised to quarantine. However, some of these individuals may be at high risk for serious illness or mortality, justifying a more complete clinical evaluation incorporating vital signs, laboratory tests and/or imaging. As primary and booster vaccines and therapeutic options may be in limited supply, the VACO Index could help health systems and governments develop a data-driven approach to allocating resources to those at greatest risk of mortality who are most likely to benefit.
Minority populations experienced a greater burden of infection in the beginning of the pandemic,19 leading to suggestions that overall population risk may seem a more equitable criterion for vaccine prioritisation. However, modelling population risk of mortality from COVID-19 encompasses two distinct components: probability of infection (infection detection dependent on access to testing and probability of a positive test) and probability of dying once infected. The associations between risk factors such as age, race and ethnicity are very different for being tested, testing positive and mortality.19 The geographic distribution of the pandemic has shifted rapidly over time, rendering testing positive a volatile outcome that has fluctuated with local disease prevalence. Thus, we argue that the best strategy is to prioritise those at greatest risk of mortality if infected.
Given the debate over the roles of BMI and race/ethnicity in COVID-19 outcomes, we carefully investigated their associations with mortality in our data. Including BMI in the Index did not improve the discrimination of the Index in VA or YNHH data. US and European studies have shown conflicting results about the relationship between BMI and COVID-19 mortality.20–24 Adding race/ethnicity to the VACO Index did not improve the accuracy of the predictions. Further, associations with race/ethnicity varied among the cohorts, with Black race and Hispanic ethnicity associated with reduced risk of mortality in VA and YNHH data and increased risk among Medicare patients. Since race and ethnicity are largely proxies for socioeconomic disparities,25 it is not surprising that their association with mortality among those testing positive for SARS-CoV-2 is variable. National VA data frequently demonstrate fewer disparities by race in health outcomes than outside VA, likely from reduced economic barriers to care and national efforts to ensure quality of care.26 27 While we considered offering an adjustment for race/ethnicity to be applied outside VA, we are concerned that adding race/ethnicity would introduce substantial noise in the risk estimation due to variation in association across facilities and would confuse biology and socioeconomics.25 28 Importantly, the VACO Index demonstrated consistent or better accuracy in racial and ethnic minority patients than in Non-Hispanic White patients.
The VACO Index modestly overestimated mortality in VA and Medicare populations and underestimated in the YNHH population. The Index was developed using patients diagnosed early in the pandemic, and improvements in care over time have decreased COVID-19 mortality.29 YNHH data had incomplete capture of comorbidities for some patients with limited healthcare system contact prior to infection, resulting in a modest underestimation of mortality. Nevertheless, the calibration plots show a monotonic increase in mortality with higher VACO Index scores, identifying patients at greatest risk of death should they become infected, and thus who should be prioritised for primary and booster vaccinations and greater clinical attention or scarce treatments.
The discrimination of the VACO Index quantified by AUC in Medicare patients was lower than in VA or YNHH patients overall. This discrepancy is related to the reciprocal relationship between the proportion of outcome events in a population and AUC—all else equal, the AUC will be lower in populations where the outcome is more common, or when a major driver of risk (ie, age) has a reduced range.30 31 Importantly, the AUC was highly consistent in patients 65 years and over across VA and Medicare patients, and calibration curves in Medicare data demonstrated good agreement between observed and expected mortality rates across race/ethnicity.
Our goal was to develop and validate a short-term COVID-19 mortality index using preinfection data—demographics and comorbidities—to inform patient, provider, healthcare organisations and government decision-making about social distancing, masking, additional testing, primary and booster vaccinations and allocation of scarce resources. Most published validated COVID-19 mortality risk indices use clinical information collected at the time of diagnosis—symptoms, vital signs, laboratory findings, imaging—that may be already influenced by the infection, and thus are not comparable to the VACO Index.8 32–37 Others appear to have similar or superior performance to the VACO Index, but have a more limited denominator (eg, only inpatients38), a different outcome (eg, mortality data limited to inpatient stay,6 a combined outcome of infection and mortality7) or were not validated in a nationwide data set.6 9 10 39
There are limitations to our study. While we were able to evaluate the index in a US national sample of over 440 000 people, most of our patients were over 50 years old, limiting our ability to validate in younger patients. Our calibration is based on mortality events occurring before August 2020 when mostly symptomatic patients were being tested. The VACO Index may overestimate risk of mortality among asymptomatic patients, and this will require further study. Nevertheless, the VACO Index represents the most widely validated risk index for short-term mortality from COVID-19. This standardised and validated index can supplement clinical judgement to help inform patient management and health policy.
What is already known on this subject
Many COIVD-19 mortality indices use complex clinical, laboratory or imaging data that are often not readily available or analysable, concatenate COVID-19 infectivity and mortality, have poor discrimination or are derived from a regional population.
A generalisable COVID-19 mortality risk index based on demographics and pre-existing conditions could inform patients and providers considering COVID-19 mitigation, prevention and treatment options even before patients present to a healthcare system.
The Veterans Health Administration COVID-19 (VACO) Index, developed and validated in a nationwide US veteran population, predicts COVID-19 30-day mortality based on data directly accessible in the electronic health record: age, sex and pre-existing medical conditions.
What this study adds
We assessed the generalisability of the VACO Index by assessing its accuracy in a US academic medical centre inpatient population and a nationwide US Medicare inpatient and outpatient cohort (aged 65 and older).
The VACO Index modestly overestimates risk in Veterans Health Administration and Medicare patients and underestimates risk in an academic medical centre population. The Index has consistent performance over time, and among important patient subgroups including women, Blacks, Hispanics, Asians and Native Americans. The VACO Index is generalisable and could inform primary and booster vaccination prioritisation, and among outpatients who test positive for SARS-CoV-2, indicate who should receive greater clinical attention or scarce treatments.
Data availability statement
Data may be obtained from a third party and are not publicly available. The United States Department of Veterans Affairs (VA) places legal restrictions on access to veteran’s health care data, which includes both identifying data and sensitive patient information. The analytic data sets used for this study are not permitted to leave the VA firewall without a Data Use Agreement. This limitation is consistent with other studies based on VA data. However, VA data are made freely available to researchers behind the VA firewall with an approved VA study protocol. For more information, please visit https://www.virec.research.va.gov or contact the VA Information Resource Center (VIReC) at VIReC{at}va.gov. CareJourney has an existing Data Use Agreement with CMS that prohibits disclosure of patient health care data. The datasets used for this study are not permitted to be accessed unless users have an existing Data Use Agreement with CMS for the necessary datasets. This limitation is consistent with studies done by entities with access to Medicare Claims through the CMS Innovators' License.
Ethics statements
Patient consent for publication
Ethics approval
This study was approved by the Institutional Review Boards of VA Connecticut Healthcare System and Yale University (0013 for both Institutional Review Boards). The Institutional Review Boards of VA Connecticut Healthcare System and Yale University granted a waiver of informed consent.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Conceptualisation: JTK, ACJ, NK, APC. Data curation: JTK, ZMB, CTR, APC, RCH. Formal analysis: JTK, ZMB. Funding acquisition: ACJ. Methodology: JTK, ACJ, APC, NK, JT, CTR. Project administration: ACJ, JTK. Resources: ACJ, NK, APC. Software: JPH, GAW. Supervision: JTK, ACJ. Visualisation: JTK, JSY. Writing (original draft): JTK, ACJ. Writing (review and editing): JTK, JSY, ZMB, JPH, GAW, CTR, JT, NK, RCH, APC, ACJ.
Funding This work was supported by the Department of Veterans Affairs, Office of Research and Development, Million Veteran Program Core (MVP000) (https://www.research.va.gov/) and the National Institute on Alcohol Abuse and Alcoholism (U01-AA026224, U24-AA020794, U01-AA020790, U10-AA013566) (https://www.niaaa.nih.gov/).
Disclaimer The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. The views and opinions expressed in this manuscript are those of the authors and do not necessarily represent those of the Department of Veterans Affairs or the US government.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.