Subscribe to RSS
Prediction of Emergency Department Hospital Admission Based on Natural Language Processing and Neural Networks[*]
25 February 2017
accepted in revised form: 26 July 2017
24 January 2018 (online)
Objective: To describe and compare logistic regression and neural network modeling strategies to predict hospital admission or transfer following initial presentation to Emergency Department (ED) triage with and without the addition of natural language processing elements.
Methods: Using data from the National Hospital Ambulatory Medical Care Survey (NHAMCS), a cross-sectional probability sample of United States EDs from 2012 and 2013 survey years, we developed several predictive models with the outcome being admission to the hospital or transfer vs. discharge home. We included patient characteristics immediately available after the patient has presented to the ED and undergone a triage process. We used this information to construct logistic regression (LR) and multilayer neural network models (MLNN) which included natural language processing (NLP) and principal component analysis from the patient’s reason for visit. Ten-fold cross validation was used to test the predictive capacity of each model and receiver operating curves (AUC) were then calculated for each model.
Results: Of the 47,200 ED visits from 642 hospitals, 6,335 (13.42%) resulted in hospital admission (or transfer). A total of 48 principal components were extracted by NLP from the reason for visit fields, which explained 75% of the overall variance for hospitalization. In the model including only structured variables, the AUC was 0.824 (95% CI 0.818-0.830) for logistic regression and 0.823 (95% CI 0.817-0.829) for MLNN. Models including only free-text information generated AUC of 0.742 (95% CI 0.7310.753) for logistic regression and 0.753 (95% CI 0.742-0.764) for MLNN. When both structured variables and free text variables were included, the AUC reached 0.846 (95% CI 0.839-0.853) for logistic regression and 0.844 (95% CI 0.836-0.852) for MLNN.
Conclusions: The predictive accuracy of hospital admission or transfer for patients who presented to ED triage overall was good, and was improved with the inclusion of free text data from a patient’s reason for visit regardless of modeling approach. Natural language processing and neural networks that incorporate patient-reported outcome free text may increase predictive accuracy for hospital admission.
KeywordsNatural language processing - neural networks - logistic regression - prediction - emergency department
* Supplementary material published on our website https://doi.org/10.3414/ME17-01-0024
- 1 Sun Y, Heng BH, Tay SY, Seow E. Predicting hospital admissions at emergency department triage using routine administrative data. Academic Emergency Medicine 2011; 18 (08) 844-850.
- 2 Handly N, Thompson DA, Li J, Chuirazzi DM, Venkat A. Evaluation of a hospital admission prediction model adding coded chief complaint data using neural network methodology. European Journal of Emergency Medicine 2015; 22 (02) 87-91.
- 3 Golmohammadi D. Predicting hospital admissions to reduce emergency department boarding. International Journal of Production Economics 2016; 182: 535-544.
- 4 Future of Emergency Care: Hospital-Based Emergency Care at the Breaking Point. Washington, DC: Institute of Medicine; 2006
- 5 Sun BC, Hsia RY, Weiss RE. et al. Effect of emergency department crowding on outcomes of admitted patients. Annals of Emergency Medicine 2013; 61 (06) 605-611. e606.
- 6 Pines JM, Hilton JA, Weber EJ. et al. International perspectives on emergency department crowding. Academic Emergency Medicine 2011; 18 (12) 1358-1370.
- 7 Chalfin DB, Trzeciak S, Likourezos A, Baumann BM, Dellinger RP, group D-Es. Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Critical Care Medicine 2007; 35 (06) 1477-1483.
- 8 Rabin E, Kocher K, McClelland M. et al. Solutions to emergency department ‘boarding’ and crowding are underused and may need to be legislated. Health Affairs 2012; 31 (08) 1757-1766.
- 9 Forero R, McCarthy S, Hillman K. Access block and emergency department overcrowding. Critical Care 2011; 15 (02) 1.
- 10 Peck JS, Benneyan JC, Nightingale DJ, Gaehde SA. Predicting emergency department inpatient admissions to improve same-day patient flow. Academic Emergency Medicine 2012; 19 (09) E1045-1054.
- 11 Leegon J, Jones I, Lanaghan K, Aronsky D. Predicting hospital admission in a pediatric Emergency Department using an Artificial Neural Network. AMIA Annual Symposium Proceedings. 2006: 1004.
- 12 McCallum A. Information extraction: Distilling structured data from unstructured text. Queue 2005; 3 (09) 48-57.
- 13 Cohen AM, Hersh WR. A survey of current work in biomedical text mining. Briefings in Bioin- formatics 2005; 6 (01) 57-71.
- 14 Claster W, Shanmuganathan S, Ghotbi N. Text mining of medical records for radiodiagnostic decision-making. Journal of Computers 2008; 3 (01) 1-6.
- 15 Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support?. Journal of Biomedical Informatics 2009; 42 (05) 760-772.
- 16 Lin C, Karlson EW, Canhao H. et al. Automatic prediction of rheumatoid arthritis disease activity from the electronic medical records. PLoS One 2013; 8 (08) e69932.
- 17 Sadat-Hashemi SM, Kazemnejad A, Lucas C, Badie K. Predicting the type of pregnancy using artificial neural networks and multinomial logistic regression: a comparison study. Neural Computing & Applications 2005; 14 (03) 198-202.
- 18 Penny W, Frost D. Neural networks in clinical medicine. Medical Decision Making 1996; 16 (04) 386-398.
- 19 Heckerling PS, Gerber BS, Tape TG, Wigton RS. Prediction of community-acquired pneumonia using artificial neural networks. Medical Decision Making 2003; 23 (02) 112-121.
- 20 Lammers RL, Hudson DL, Seaman ME. Prediction of traumatic wound infection with a neural network-derived decision model. American journal of Emergency Medicine 2003; 21 (01) 1-7.
- 21 Launay C, Rivière H, Kabeshova A, Beauchet O. Predicting prolonged length of hospital stay in older emergency department users: Use of a novel analysis method, the Artificial Neural Network. European Journal of Internal Medicine 2015; 26 (07) 478-482.
- 22 Hess EP, Wells GA, Jaffe A, Stiell IG. A study to derive a clinical decision rule for triage of emergency department patients with chest pain: design and methodology. BMC Emergency Medicine 2008; 8: 3.
- 23 Green M, Björk J, Forberg J, Ekelund U, Edenbrandt L, Ohlsson M. Comparison between neural networks and multiple logistic regression to predict acute coronary syndrome in the emergency room. Artificial Intelligence in Medicine 2006; 38 (03) 305-318.
- 24 Carlson JN, Das S, De la Torre F. et al. A Novel Artificial Intelligence System for Endotracheal Intubation. Prehospital Emergency Care. 2016: 1-5.
- 25 McCaig LF, Burt CW. Understanding and Interpreting the National Hospital Ambulatory Medical Care Survey: Key Questions and Answers. Annals of Emergency Medicine 2012; 60 (06) 716-721.
- 26 Lum HD, Studenski SA, Degenholtz HB, Hardy SE. Early hospital readmission is a predictor of one-year mortality in community-dwelling older Medicare beneficiaries. J Gen Intern Med 2012; 27 (11) 1467-1474.
- 27 Centers for Disease Control and Prevention NHAMCS micro-data file documentation [cited 2017 Jan 19]. 2013 Available from: ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NHAMCS/doc13_ed.pdf.
- 28 Stoltzfus JC. Logistic regression: a brief primer. Academic Emergency Medicine 2011; 18 (10) 1099-1104.
- 29 Zhang X, Liu Y, Yang M, Zhang T, Young AA, Li X. Comparative study of four time series methods in forecasting typhoid fever incidence in China. PLoS One 2013; 8 (05) e63116.
- 30 Zhao Z, Xin H, Ren Y, Guo X. Application and comparison of BP neural network algorithm in MATLAB. International Conference on Measuring Technology and Mechatronics Automation (ICMTMA). 2010
- 31 Vijayarani S, Ilamathi MJ, Nithya M. Preprocessing Techniques for Text Mining-An Overview. International Journal of Computer Science and Communication Networks 2015; 5 (01) 7-16.
- 32 Zhang Y, Jin R, Zhou Z-H. Understanding bag-of- words model: a statistical framework. International Journal of Machine Learning and Cybernetics 2010; 1 1-4 43-52.
- 33 Zhang X, Cowan BR, Bluemke DA. et al. Atlas- based quantification of cardiac remodeling due to myocardial infarction. PLoS One 2014; 9 (10) e110243.
- 34 Zhang X, Ambale-Venkatesh B, Bluemke DA. et al. Information maximizing component analysis of left ventricular remodeling due to myocardial infarction. Journal of Translational Medicine 2015; 13 (01) 1.
- 35 Leegon J, Jones I, Lanaghan K, Aronsky D. Predicting hospital admission for Emergency Department patients using a Bayesian network. AMIA Annual Symposium Proceedings. 2005: 1022.
- 36 Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N. Reliability and validity of a new five-level triage instrument. Academic Emergency Medicine 2000; 7 (03) 236-242.
- 37 Botsis T, Nguyen MD, Woo EJ, Markatou M, Ball R. Text mining for the Vaccine Adverse Event Reporting System: medical text classification using informative feature selection. Journal of the American Medical Informatics Association 2011; 18 (05) 631-638.
- 38 Husk G, Akhtar S. Chief complaints, emergency department clinical documentation systems, and the challenge of dealing with the patient’s own words. Academic Emergency Medicine 2007; 14 (01) 69-73.
- 39 Thompson DA, Yarnold PR, Williams DR, Adams SL. Effects of actual waiting time, perceived waiting time, information delivery, and expressive quality on patient satisfaction in the emergency department. Annals of Emergency Medicine 1996; 28 (06) 657-665.
- 40 Day FC, Schriger DL, La M. Automated linking of free-text complaints to reason-for-visit categories and International Classification of Diseases diagnoses in emergency department patient record databases. Annals of Emergency Medicine 2004; 43 (03) 401-409.
- 41 Dybowski R, Gant V. Clinical applications of artificial neural networks. Cambridge: Cambridge University Press; 2001
- 42 Lo-Ciganic WH, Donohue JM, Thorpe JM. et al. Using machine learning to examine medication adherence thresholds and risk of hospitalization. Medical Care 2015; 53 (08) 720-728.
- 43 Becalick DC, Coats TJ. Comparison of artificial intelligence techniques with UKTRISS for estimating probability of survival after trauma. UK Trauma and Injury Severity Score. The Journal of Trauma 2001; 51 (01) 123-133.
- 44 Hsich E, Gorodeski EZ, Blackstone EH, Ishwaran H, Lauer MS. Identifying important risk factors for survival in patient with systolic heart failure using random survival forests. Circulation Cardiovascular Quality and Outcomes 2011; 4 (01) 39-45.
- 45 Parsaeian M, Mohammad K, Mahmoudi M, Zeraati H. Comparison of logistic regression and artificial neural network in low back pain prediction: second national health survey. Iranian Journal of Public Health 2012; 41 (06) 86.
- 46 Song JH, Venkatesh SS, Conant EA, Arger PH, Sehgal CM. Comparative analysis of logistic regression and artificial neural network for computer-aided diagnosis of breast masses. Academic Radiology 2005; 12 (04) 487-495.
- 47 Forberg JL, Green M, Bj ork J. et al. In search of the best method to predict acute coronary syndrome using only the electrocardiogram from the emergency department. Journal of Electrocardiology 2009; 42 (01) 58-63.
- 48 Gholipour C, Rahim F, Fakhree A, Ziapour B. Using an Artificial Neural Networks (ANNs) Model for Prediction of Intensive Care Unit (ICU) Outcome and Length of Stay at Hospital in Traumatic Patients. Journal of Clinical and Diagnostic Research 2015; 9 (04) OC19-23.
- 49 Tsai PF, Chen PC, Chen YY. et al. Length of Hospital Stay Prediction at the Admission Stage for Cardiology Patients Using Artificial Neural Network. Journal of Healthcare Engineering 2016; 2016: 7035463.
- 50 Sinha M, Khor KN, Amresh A, Drachman D, Frechette A. The use of a kiosk-model bilingual self- triage system in the pediatric emergency department. Pediatr Emerg Care 2014; 30 (01) 63-68.
- 51 Jaimes F, Farbiarz J, Alvarez D, Martínez C. Comparison between logistic regression and neural networks to predict death in patients with suspected sepsis in the emergency room. Critical Care 2005; 9 (02) 1.
- 52 Bartfay E, Mackillop W, Pater J. Comparing the predictive value of neural network models to logistic regression models on the risk of death for small cell lung cancer patients. European Journal of Cancer Care 2006; 15 (02) 115-124.
- 53 Dexheimer JW, Leegon J, Aronsky D. Predicting hospital admission at triage in an emergency department. AMIA Annual Symposium Proceedings. 2007: 937.
- 54 Tanabe P, Gimbel R, Yarnold PR, Adams JG. The Emergency Severity Index (version 3) 5-level triage system scores predict ED resource consumption. J Emerg Nurs 2004; 30 (01) 22-29.
- 55 McHugh M, Tanabe P, McClelland M, Khare RK. More patients are triaged using the Emergency Severity Index than any other triage acuity system in the United States. Academic Emergency Medicine 2012; 19 (01) 106-109.
- 56 Wuerz R. Emergency severity index triage category is associated with six-month survival. ESI Triage Study Group. Academic Emergency Medicine 2001; 8 (01) 61-64.
- 57 Hirshon JM, Warner M, Irvin CB. et al. Research using emergency department-related data sets: current status and future directions. Academic Emergency Medicine 2009; 16 (11) 1103-1109.
- 58 Roberts A, Gaizauskas R, Hepple M. et al. The CLEF corpus: semantic annotation of clinical text. AMIA Annual Symposium Proceedings 2007; 2007: 625-629.
- 59 Gurulingappa H, Rajput AM, Roberts A, Fluck J, Hofmann-Apitius M, Toldo L. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics 2012; 45 (05) 885-892.
- 60 Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Research 2004; 32 (Suppl. 01) D267-D270.
- 61 Wallach HM. Topic modeling: beyond bag-of- words. Proceedings of the 23rd International Conference on Machine Learning. 2006
- 62 Newman D, Karimi S, Cavedon L. Using Topic Models to Interpret MEDLINE’s Medical Subject Headings. Proceedings of the 22nd Australasian Joint Conference on Advances in Artificial Intelligence (AI ‘09).. Berlin, Heidelberg: Springer; 2009: 270-279.
- 63 Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: an introduction. Journal of the American Medical Informatics Association 2011; 18 (05) 544-551.
- 64 Yu W, Liu T, Valdez R, Gwinn M, Khoury MJ. Application of support vector machine modeling for prediction of common diseases: the case of diabetes and pre-diabetes. BMC Medical Informatics and Decision Making 2010; 10 (01) 16.
- 65 Özçift A. Random forests ensemble classifier trained with data resampling strategy to improve cardiac arrhythmia diagnosis. Computers in Biology and Medicine 2011; 41 (05) 265-271.
- 66 Subasi A. Automatic recognition of alertness level from EEG by using neural network and wavelet coefficients. Expert Systems with Applications 2005; 28 (04) 701-711.
- 67 Bar Y, Diamant I, Wolf L, Greenspan H. Deep learning with non-medical training used for chest pathology identification. SPIE Medical Imaging. 2015: 94140V.