Subscribe to RSS
DOI: 10.1055/a-2648-4817
Implementation of Passive Deterioration Index Alerts in an Intermediate Care Unit: A Failed Early Warning System Strategy
Funding This study was supported by grant P30HS029744 from the Agency for Healthcare Research and Quality (AHRQ). The University of Minnesota Office of Academic Clinical Affairs, Clinical Translational Science Institute, and Center for Learning Health System Sciences offered additional support through the Minnesota Learning Health System (MN-LHS) Mentored Career Development Program. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ or MN-LHS.

Abstract
Objectives
Traditional early warning systems (EWS) have shown uncertain efficacy in real-world settings. More recently, machine learning models like the Epic Deterioration Index (DTI) have been developed, promising greater accuracy. Recognizing the potential of DTI, but also the pervasive issue of alert fatigue with interruptive (i.e., pop-up) EWS alerts, our institution implemented a DTI-enabled EWS with passive alerts (colored icons visible in prespecified locations within the electronic health record). We hypothesized that our intervention would reduce the time to treatment for deteriorating patients.
Methods
We piloted a DTI-enabled EWS in a 30-bed intermediate care unit at a large academic medical center. DTI scores, alert icons, and vital signs appeared on a custom Patient List interface. In the event of an alert, charge nurses were expected to conduct a bedside assessment and escalate care as necessary. We compared the 111-day pre- and postimplementation periods, with alert-to-action time as the primary outcome. Secondary outcomes included mortality, length of stay, ICU transfer, documentation rate, and provider acceptance.
Results
Among 301 patients with an elevated-risk score (156 pre- and 145 postimplementation), we found no significant differences in alert-to-action time (469 vs. 359 minutes before alert; p = 0.96), with provider actions typically occurring several hours before the alert in both periods. There were no significant differences in mortality (10.3% vs. 13.1%; p = 0.56), length of stay (15.7 vs. 12.8 days; p = 0.23), or ICU transfer (8.3% vs. 6.2%; p = 0.63). Charge nurses documented acknowledgment of the alert in 18.6% of cases, and acceptance was poor. Most nurses expressed a preference for interruptive alerts and more prominent DTI display locations.
Conclusion
In this single-unit pilot, passive DTI-enabled EWS alerts did not improve time to intervention or clinical outcomes. High-risk DTI scores often occurred after clinical deterioration had already been recognized.
Keywords
severity of illness index - machine learning - decision support systems - length of stay - mortalityProtection of Human and Animal Subjects
The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the University of Minnesota Institutional Review Board.
Publication History
Received: 10 February 2025
Accepted: 03 July 2025
Article published online:
27 August 2025
© 2025. Thieme. All rights reserved.
Georg Thieme Verlag KG
Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany
-
References
- 1 McGaughey J, Fergusson DA, Van Bogaert P, Rose L. Early warning systems and rapid response systems for the prevention of patient deterioration on acute adult hospital wards. Cochrane Database Syst Rev 2021; 11 (11) CD005529
- 2 Epic Systems Corporation. Cognitive Computing Model Brief: Deterioration Index. Accessed April 30, 2025 at: https://galaxy.epic.com/?#Browse/page=1!68!50!3883949
- 3 Artificial Intelligence Triggers Fast. Lifesaving Care for COVID-19 Patients. Accessed April 30, 2025 at: https://www.epic.com/epic/post/artificial-intelligence-epic-triggers-fast-lifesaving-care-covid-19-patients
- 4 Cacciaglia A. Saving Lives with AI: Using the Deterioration Index Predictive Model to Help Patients Sooner. EpicShare. Accessed April 30, 2025 at: https://www.epicshare.org/share-and-learn/saving-lives-with-ai
- 5 Herasevich S, Lipatov K, Pinevich Y. et al. The impact of health information technology for early detection of patient deterioration on mortality and length of stay in the hospital acute care setting: systematic review and meta-analysis. Crit Care Med 2022; 50 (08) 1198-1209
- 6 Blythe R, Parsons R, White NM, Cook D, McPhail S. A scoping review of real-time automated clinical deterioration alerts and evidence of impacts on hospitalised patient outcomes. BMJ Qual Saf 2022; 31 (10) 725-734
- 7 Braun EJ, Singh S, Penlesky AC. et al. Nursing implications of an early warning system implemented to reduce adverse events: a qualitative study. BMJ Qual Saf 2022; 31 (10) 716-724
- 8 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
- 9 Chaparro JD, Beus JM, Dziorny AC. et al. Clinical decision support stewardship: best practices and techniques to monitor and improve interruptive alerts. Appl Clin Inform 2022; 13 (03) 560-568
- 10 Byrd IV TF, Southwell B, Ravishankar A. et al. Validation of a proprietary deterioration index model and performance in hospitalized adults. JAMA Netw Open 2023; 6 (07) e2324176
- 11 Salehinejad H, Meehan AM, Rahman PA, Core MA, Borah BJ, Caraballo PJ. Novel machine learning model to improve performance of an early warning system in hospitalized patients: a retrospective multisite cross-validation study. EClinicalMedicine 2023; 66: 102312
- 12 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
- 13 Bedoya AD, Clement ME, Phelan M, Steorts RC, O'Brien C, Goldstein BA. Minimal impact of implemented early warning score and best practice alert for patient deterioration. Crit Care Med 2019; 47 (01) 49-55
- 14 Ginestra JC, Giannini HM, Schweickert WD. et al. Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock. Crit Care Med 2019; 47 (11) 1477-1484
- 15 Li RC, Smith M, Lu J. et al. Using AI to empower collaborative team workflows: two implementations for advance care planning and care escalation. NEJM Catal 2022; 3 (04) CAT.21.0457
- 16 Blythe R, Naicker S, White N. et al. Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care. BMC Med Inform Decis Mak 2024; 24 (01) 241
- 17 Macheel C, Reicks P, Sybrant C. et al. Clinical decision support intervention for rib fracture treatment. J Am Coll Surg 2020; 231 (02) 249-256.e2
- 18 Byrd IV TF, Kim JS, Yeh C, Lee J, O'Leary KJ. Technology acceptance and critical mass: development of a consolidated model to explain the actual use of mobile health care communication tools. J Biomed Inform 2021; 117: 103749
- 19 Venkatesh V, Thong JYL, Xu X. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. Manage Inf Syst Q 2012; 36 (01) 157-178
- 20 Edelson DP, Churpek MM, Carey KA. et al. Early warning scores with and without artificial intelligence. JAMA Netw Open 2024; 7 (10) e2438986
- 21 Winter MC, Kubis S, Bonafide CP. Beyond reporting early warning score sensitivity: the temporal relationship and clinical relevance of “true positive” alerts that precede critical deterioration. J Hosp Med 2019; 14 (03) 138-143
- 22 Kamran F, Tang S, Otles E. et al. Early identification of patients admitted to hospital for COVID-19 at risk of clinical deterioration: model development and multisite external validation study. BMJ 2022; 376: e068576
- 23 Beaulieu-Jones BK, Yuan W, Brat GA. et al. Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians? npj. Digit Med 2021; 4 (01) 1-6
- 24 van der Vegt AH, Campbell V, Mitchell I. et al. Systematic review and longitudinal analysis of implementing artificial intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inform Assoc 2024; 31 (02) 509-524
- 25 Zebehazy S, Ritter C, Larson C, Bowles J. Validating and Implementing a Deterioration Index Support Tool: A Proactive Approach to Reduce Escalations of Care. 2024 ;9(2)
- 26 Valvona SN, Rayo MF, Abdel-Rasoul M. et al. Comparative Effectiveness of Best Practice Alerts with Active and Passive Presentations: A Retrospective Study. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care. 2020 ;9(1): 105-109
- 27 Long D, Capan M, Mascioli S, Weldon D, Arnold R, Miller K. Evaluation of user-interface alert displays for clinical decision support systems for sepsis. Crit Care Nurse 2018; 38 (04) 46-54
- 28 Murad DA, Tsugawa Y, Elashoff DA, Baldwin KM, Bell DS. Distinct components of alert fatigue in physicians' responses to a noninterruptive clinical decision support alert. J Am Med Inform Assoc 2022; 30 (01) 64-72
- 29 Gallo RJ, Shieh L, Smith M. et al. Effectiveness of an artificial intelligence-enabled intervention for detecting clinical deterioration. JAMA Intern Med 2024; 184 (05) 557-562
- 30 Evans RS, Kuttler KG, Simpson KJ. et al. Automated detection of physiologic deterioration in hospitalized patients. J Am Med Inform Assoc 2015; 22 (02) 350-360
- 31 Bailey TC, Chen Y, Mao Y. et al. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med 2013; 8 (05) 236-242
- 32 Bhavani SV. A call for a consensus approach to the design, implementation, and evaluation of early warning systems. Crit Care Med 2022; 50 (08) 1280-1282
- 33 Shashikumar SP, Le JP, Yung N. et al. Development and validation of a deep learning model for prediction of adult physiological deterioration. Crit Care Explor 2024; 6 (09) e1151
- 34 Churpek MM, Ingebritsen R, Carey KA. et al. Causes, diagnostic testing, and treatments related to clinical deterioration events among high-risk ward patients. Crit Care Explor 2024; 6 (10) e1161
- 35 Byrd IV TF, Phelan TA, Ingraham NE. et al. Beyond unplanned ICU transfers: linking a revised definition of deterioration to patient outcomes. Crit Care Med 2024; 52 (09) e439-e449
- 36 Korach ZT, Cato KD, Collins SA. et al. Unsupervised machine learning of topics documented by nurses about hospitalized patients prior to a rapid-response event. Appl Clin Inform 2019; 10 (05) 952-963
- 37 Rossetti SC, Dykes PC, Knaplund C. et al. Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial. Nat Med 2025; 31 (06) 1895-1902
- 38 Escobar GJ, Liu VX, Schuler A, Lawson B, Greene JD, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med 2020; 383 (20) 1951-1960
- 39 Singh S, Laud PW, Crotty BH. et al. Effect of implementing a commercial electronic early warning system on outcomes of hospitalized patients. Am J Med Qual 2023; 38 (05) 229-237
- 40 Tulk Jesso S, Kelliher A, Sanghavi H, Martin T, Henrickson Parker S. Inclusion of clinicians in the development and evaluation of clinical artificial intelligence tools: a systematic literature review. Front Psychol 2022; 13: 830345
- 41 Kang MJ, Rossetti SC, Lowenthal G. et al. Designing and testing clinical simulations of an early warning system for implementation in acute care settings. JAMIA Open 2024; 7 (04) ooae092
- 42 Kandaswamy S, Williams H, Thompson SA, Dawson TE, Muthu N, Orenstein EW. Realizing the full potential of clinical decision support: translating usability testing into routine practice in health care operations. Appl Clin Inform 2024; 15 (05) 1039-1048