Development of a prediction model for 30-day COVID-19 hospitalization and death in a national cohort of Veterans Health Administration patients–March 2022—April 2023
This study addresses the need for updated COVID-19 risk prediction models that reflect the current context of the pandemic, including the Omicron variant, population immunity from prior infection and vaccination, and the availability of effective treatments. Previous models developed early in the pandemic may no longer be accurate or applicable due to these evolving factors.
College of Health researcher(s)
Abstract
Objective
The epidemiology of COVID-19 has substantially changed since its emergence given the availability of effective vaccines, circulation of different viral variants, and re-infections. We aimed to develop models to predict 30-day COVID-19 hospitalization and death in the Omicron era for contemporary clinical and research applications.
Methods
We used comprehensive electronic health records from a national cohort of patients in the Veterans Health Administration (VHA) who tested positive for SARS-CoV-2 between March 1, 2022, and March 31, 2023. Full models incorporated 84 predictors, including demographics, comorbidities, and receipt of COVID-19 vaccinations and anti-SARS-CoV-2 treatments. Parsimonious models included 19 predictors. We created models for 30-day hospitalization or death, 30-day hospitalization, and 30-day all-cause mortality. We used the Super Learner ensemble machine learning algorithm to fit prediction models. Model performance was assessed with the area under the receiver operating characteristic curve (AUC), Brier scores, and calibration intercepts and slopes in a 20% holdout dataset.
Results
Models were trained and tested on 198,174 patients, of whom 8% were hospitalized or died within 30 days of testing positive. AUCs for the full models ranged from 0.80 (hospitalization) to 0.91 (death). Brier scores were close to 0, with the lowest error in the mortality model (Brier score: 0.01). All three models were well calibrated with calibration intercepts <0.23 and slopes <1.05. Parsimonious models performed comparably to full models.
Conclusions
We developed prediction models that accurately estimate COVID-19 hospitalization and mortality risk following emergence of the Omicron variant and in the setting of COVID-19 vaccinations and antiviral treatments. These models may be used for risk stratification to inform COVID-19 treatment and to identify high-risk patients for inclusion in clinical trials.
COVID-19 Hospitalization and Death Risk Prediction Model FAQ
What is the purpose of this study?
This study aimed to develop and validate prediction models for estimating the risk of COVID-19 hospitalization and death among patients within the Veterans Health Administration (VHA) system. These models were created using data from the Omicron era and incorporate information on COVID-19 vaccinations and antiviral treatments, making them relevant to the current landscape of the pandemic.
Who are the models intended for?
These models are intended for use within the VHA healthcare system to help healthcare providers identify patients at high risk of hospitalization or death from COVID-19. This information can be used to inform treatment decisions, prioritize care, and potentially identify patients suitable for inclusion in clinical trials. Additionally, a simplified version of the model (the "parsimonious model") was developed for potential use in non-VHA healthcare systems.
What data was used to develop the models?
The models were developed using comprehensive electronic health records from a national cohort of 198,174 VHA patients who tested positive for SARS-CoV-2 between March 2022 and April 2023. The data includes demographics, comorbidities, COVID-19 vaccination status, and receipt of outpatient anti-SARS-CoV-2 treatments.
What are the key predictors of COVID-19 hospitalization and death?
The most important predictor identified was the Care Assessment Needs (CAN) score, a VHA-specific score that predicts 1-year mortality risk. Other important predictors included age, measures of frailty (Charlson Comorbidity Index and Area Deprivation Index), cardiovascular disease, COVID-19 vaccination status, and receipt of outpatient COVID-19 treatments.
How well do the models perform?
The models demonstrated good discrimination and calibration in internal validation. The Area Under the Receiver Operating Characteristic Curve (AUC) values, a measure of discrimination, ranged from 0.80 to 0.91, indicating good performance. The models also exhibited good calibration, meaning that the predicted risks aligned well with the observed risks.
How do these models compare to existing risk scores?
The developed models outperformed existing risk scores, such as the VACO Index and the CAN score, in terms of discrimination, calibration, and classification accuracy for predicting COVID-19 mortality. This improvement likely stems from the inclusion of more recent data reflecting the current context of the pandemic, including vaccination and treatment availability.
Are there any limitations to these models?
The models have some limitations. Data on re-infections was not available, which is an important factor for predicting severe outcomes. Additionally, while a wide range of predictors were considered, it is possible that unmeasured prognostic factors could influence risk. Finally, the models need to be externally validated in other populations and healthcare systems.
How can these models be used in clinical practice and research?
Within the VHA, the models can be used to automate risk scores for patients, similar to the CAN score. This can facilitate targeted interventions, such as prescribing outpatient treatments to high-risk individuals. The simplified "parsimonious" models could be implemented in other healthcare systems and aid in identifying high-risk patients for clinical trials.