Performance Degradation between Development and Deployment of a Predictive Model for Central Line-Associated Bloodstream Infections in Hospitalized Children

Jonathan M. Beus; Mark Mai; Nikolay P. Braykov; Swaminathan Kandaswamy; Edwin Ray; David B. Cundiff; Paulette Djachechi; Sarah Thompson; Azade Tabaie; Ryan Birmingham; Rishi Kamaleswaran; Evan Orenstein

doi:10.1055/a-2605-1847

Applied Clinical Informatics, Table of Contents

Appl Clin Inform 2025; 16(04): 1192-1199
DOI: 10.1055/a-2605-1847

Special Issue on CDS Failures

Performance Degradation between Development and Deployment of a Predictive Model for Central Line-Associated Bloodstream Infections in Hospitalized Children

Authors

Jonathan M. Beus

¹Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Mark Mai

¹Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Nikolay P. Braykov

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Swaminathan Kandaswamy

¹Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
Edwin Ray

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
David B. Cundiff

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Paulette Djachechi

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Sarah Thompson

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
Azade Tabaie

³MedStar Health Research Institute, Center for Biostatistics, Informatics, and Data Science, Washington, DC, United States
Ryan Birmingham

⁴Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States
Rishi Kamaleswaran

⁴Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia, United States

⁵Departments of Biomedical Engineering and Electrical & Computer Engineering, Duke University, Durham, North Carolina, United States
Evan Orenstein

¹Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States

²Department of Information Systems and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States

Abstract

Background

Central line-associated bloodstream infections (CLABSIs) are associated with substantial pediatric morbidity and mortality. The capacity to predict which children with central lines are at greatest risk of CLABSI could inform surveillance and prevention efforts. Our team previously published in silico predictive models for CLABSI.

Objective

To prospectively implement a pediatric CLABSI predictive model and achieve adequate performance in offline validation for implementation in clinical practice.

Methods

Most performant predictive models were deep learning models requiring substantial pre-processing of many features into 8-hour windows including the current day and up to 56 days prior for the current admission. To replicate this pre-processing, we created a novel infrastructure to (1) organize current-day data for all the relevant features and (2) create a staged historical data store for those same features with application programming interfaces to connect the two. We compared predictive performance of these scores for CLABSI in the next 48 hours with two labels, one based on manual review of positive blood cultures in children with central lines and another based on positive blood culture and receipt of at least 4 days of new IV antibiotics.

Results

The area under the receiver-operating characteristic (AUROC) fell from 0.97 from retrospective data to <0.60 despite multiple iterations of troubleshooting. Primary root causes included train/serve skew, feature leakage, and overfitting. Hypothesized secondary drivers were complex model specification, poor data governance, inadequate testing, challenging feature translation between real-time and historical data models, limited monitoring and logging infrastructure for troubleshooting, and suboptimal handoff between the model development and deployment teams.

Conclusion

Bridging the gap from predictive model development to clinical deployment requires early and close coordination between data governance, data science, clinical informatics, and implementation engineers. Balancing predictive performance with implementation feasibility can accelerate the adoption of predictive clinical decision support systems.

Keywords

artificial intelligence - electronic health records and systems - clinical decision support - implementation and deployment - deep learning

Full Text

References

References
1 CDC. Current HAI Progress Report. Healthcare-Associated Infections (HAIs). November 25, 2024. Accessed December 29, 2024 at: https://www.cdc.gov/healthcare-associated-infections/php/data/progress-report.html
2 Guidelines for the Prevention of Intravascular Catheter-Related Infections. Accessed December 29, 2024 at: https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5110a1.htm
3 Karagiannidou S, Kourlaba G, Zaoutis T, Maniadakis N, Papaevangelou V. Attributable mortality for pediatric and neonatal central line-associated bloodstream infections in Greece. J Pediatr Intensive Care 2021; 13 (02) 174-183
4 Goudie A, Dynan L, Brady PW, Rettiganti M. Attributable cost and length of stay for central line-associated bloodstream infections. Pediatrics 2014; 133 (06) e1525-e1532
5 Bundy DG, Gaur AH, Billett AL, He B, Colantuoni EA, Miller MR. Children's Hospital Association Hematology/Oncology CLABSI Collaborative. Preventing CLABSIs among pediatric hematology/oncology inpatients: national collaborative results. Pediatrics 2014; 134 (06) e1678-e1685
6 Willis DN, Looper K, Malone RA. et al. Eliminating central line associated bloodstream infections in pediatric oncology patients: a quality improvement effort. Pediatr Qual Saf 2023; 8 (03) e660
7 Carter JH, Langley JM, Kuhle S, Kirkland S. Risk factors for central venous catheter-associated bloodstream infection in pediatric patients: a cohort study. Infect Control Hosp Epidemiol 2016; 37 (08) 939-945
8 Paioni P, Kuhn S, Strässle Y, Seifert B, Berger C. Risk factors for central line-associated bloodstream infections in children with tunneled central venous catheters. Am J Infect Control 2020; 48 (01) 33-39
9 Tabaie A, Orenstein EW, Nemati S. et al. Predicting presumed serious infection among hospitalized children on central venous lines with machine learning. Comput Biol Med 2021; 132: 104289
10 Tabaie A, Orenstein EW, Nemati S, Basu RK, Clifford GD, Kamaleswaran R. Deep learning model to predict serious infection among children with central venous lines. Front Pediatr 2021; 9: 726870
11 Tabaie A, Orenstein EW, Kandaswamy S, Kamaleswaran R. Integrating structured and unstructured data for timely prediction of bloodstream infection among children. Pediatr Res 2023; 93 (04) 969-975
12 Bloodstream Infection (BSI) Events. | PSC | NHSN | CDC. December 9, 2024. Accessed December 29, 2024 at: https://www.cdc.gov/nhsn/psc/bsi/index.html
13 Harris PA, Taylor R, Minor BL. et al; REDCap Consortium. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019; 95: 103208
14 Tennekes M, de Jonge E, Daas PJH. Visualizing and inspecting large datasets with tableplots. J Data Sci 2013; 11: 43-58
15 Corbin CK, Maclay R, Acharya A. et al. DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record. J Am Med Inform Assoc 2023; 30 (09) 1532-1542