Evaluating the Effect of a COVID-19 Predictive Model to Facilitate Discharge: A Randomized Controlled Trial

Abstract

Background We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown.

Objectives The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS).

Methods We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays.

Results Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location.

Conclusion An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result.

Trial registration ClinicalTrials.gov identifier: NCT04570488.

Keywords

COVID-19 - randomized controlled trial - pragmatic trial - artificial intelligence - clinical decision support

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Institutional Review Board of NYU Grossman School of Medicine.

Publication History

Received: 07 December 2021

Accepted: 17 April 2022

Article published online:
27 July 2022

• References

• 1 Liu J, Li S, Liu J. et al. Longitudinal characteristics of lymphocyte responses and cytokine profiles in the peripheral blood of SARS-CoV-2 infected patients. EBioMedicine 2020; 55: 102763
• 2 Guan WJ, Ni ZY, Hu Y. et al; China Medical Treatment Expert Group for Covid-19. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 2020; 382 (18) 1708-1720
• 3 Chen R, Liang W, Jiang M. et al; Medical Treatment Expert Group for COVID-19. Risk factors of fatal outcome in hospitalized subjects with coronavirus disease 2019 from a nationwide analysis in China. Chest 2020; 158 (01) 97-105
• 4 Huang C, Wang Y, Li X. et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020; 395 (10223): 497-506
• 5 Galloway JB, Norton S, Barker RD. et al. A clinical risk score to identify patients with COVID-19 at high risk of critical care admission or death: an observational cohort study. J Infect 2020; 81 (02) 282-288
• 6 Ji M, Yuan L, Shen W. et al. A predictive model for disease progression in non-severely ill patients with coronavirus disease 2019. Eur Respir J 2020; 56 (01) 2001234
• 7 Li Q, Zhang J, Ling Y. et al. A simple algorithm helps early identification of SARS-CoV-2 infection patients with severe progression tendency. Infection 2020; 48 (04) 577-584
• 8 Liang W, Liang H, Ou L. et al; China Medical Treatment Expert Group for COVID-19. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med 2020; 180 (08) 1081-1089
• 9 Yan L, Zhang HT, Goncalves J. et al. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell 2020; 2 (05) 283-288
• 10 Singh K, Valley TS, Tang S. et al. Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19. Ann Am Thorac Soc 2021; 18 (07) 1129-1137
• 11 Wynants L, Van Calster B, Collins GS. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369: m1328
• 12 Harish K, Zhang B, Stella P. et al. Validation of parsimonious prognostic models for patients infected with COVID-19. BMJ Health Care Inform 2021; 28 (01) e100267
• 13 Razavian N, Major VJ, Sudarshan M. et al. A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients. NPJ Digit Med 2020; 3: 130
• 14 Wiens J, Saria S, Sendak M. et al. Do no harm: a roadmap for responsible machine learning for health care. Nat Med 2019; 25 (09) 1337-1340
• 15 Drysdale E, Dolatabadi E, Chivers C. , et al. Implementing AI in Healthcare; 2019
• 16 Begg C, Cho M, Eastwood S. et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276 (08) 637-639
• 17 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. Squire 2.0 (Standards for Quality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. Am J Crit Care 2015; 24 (06) 466-473
• 18 Pinnock H, Barwick M, Carpenter CR. et al; StaRI Group. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ 2017; 356: i6795
• 19 Haynes RB, Del Fiol G, Michelson M, Iorio A. Context and approach in reporting evaluations of electronic health record-based implementation projects. Ann Intern Med 2020; 172 (Suppl. 11) S73-S78
• 20 R Core Team. R: A Language and Environment for Statistical Computing. Published online 2018. Accessed August 31, 2021 at: https://www.R-project.org/
• 21 Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J Exp Psychol Gen 2015; 144 (01) 114-126
• 22 Shcherbatykh I, Holbrook A, Thabane L, Dolovich L. COMPETE III investigators. Methodologic issues in health informatics trials: the complexities of complex interventions. J Am Med Inform Assoc 2008; 15 (05) 575-580
• 23 Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med 2019; 381 (12) 1175-1179
• 24 Rosenfeld LS, Goldmann F, Kaprio LA. Reasons for prolonged hospital stay; a study of need for hospital care. J Chronic Dis 1957; 6 (02) 141-152
• 25 Brasel KJ, Lim HJ, Nirula R, Weigelt JA. Length of stay: an appropriate quality measure?. Arch Surg 2007; 142 (05) 461-465
• 26 Garibaldi BT, Fiksel J, Muschelli J. et al. Patient trajectories among persons hospitalized for COVID-19: a cohort study. Ann Intern Med 2021; 174 (01) 33-41
• 27 Pereira NL, Ahmad F, Byku M. et al. COVID-19: understanding inter-individual variability and implications for precision medicine. Mayo Clin Proc 2021; 96 (02) 446-463
• 28 Guan WJ, Liang WH, Zhao Y. et al; China Medical Treatment Expert Group for COVID-19. Comorbidity and its impact on 1590 patients with COVID-19 in China: a nationwide analysis. Eur Respir J 2020; 55 (05) 2000547
• 29 Sanyaolu A, Okorie C, Marinkovic A. et al. Comorbidity and its impact on patients with COVID-19. SN Compr Clin Med 2020; 2 (08) 1069-1076