Can Prehospital Data Improve Early Identification of Sepsis in Emergency Department? An Integrative Review of Machine Learning Approaches

Abstract

Background Sepsis is associated with high mortality, especially during the novel coronavirus disease 2019 (COVID-19) pandemic. Along with high monetary health care costs for sepsis treatment, there is a lasting impact on lives of sepsis survivors and their caregivers. Early identification is necessary to reduce the negative impact of sepsis and to improve patient outcomes. Prehospital data are among the earliest information collected by health care systems. Using these untapped sources of data in machine learning (ML)-based approaches can identify patients with sepsis earlier in emergency department (ED).

Objectives This integrative literature review aims to discuss the importance of utilizing prehospital data elements in ED, summarize their current use in developing ML-based prediction models, and specifically identify those data elements that can potentially contribute to early identification of sepsis in ED when used in ML-based approaches.

Method Literature search strategy includes following two separate searches: (1) use of prehospital data in ML models in ED; and (2) ML models that are developed specifically to predict/detect sepsis in ED. In total, 24 articles are used in this review.

Results A summary of prehospital data used to identify time-sensitive conditions earlier in ED is provided. Literature related to use of ML models for early identification of sepsis in ED is limited and no studies were found related to ML models using prehospital data in prediction/early identification of sepsis in ED. Among those using ED data, ML models outperform traditional statistical models. In addition, the use of the free-text elements and natural language processing (NLP) methods could result in better prediction of sepsis in ED.

Conclusion This study reviews the use of prehospital data in early decision-making in ED and suggests that researchers utilize such data elements for prediction/early identification of sepsis in ML-based approaches.

Keywords

artificial intelligence - early identification - machine learning - prehospital data - sepsis

Background and Significance

Sepsis is a result of overwhelming immune response to combat an infection, leading to widespread inflammation and subsequent damage to organs and tissue impairment.[1] At least 1.7 million adults in the United States develop sepsis and of those 270,000 results in death annually.[2]

Sepsis, besides being associated with high mortality and increased lengths of intensive care unit (ICU) and hospital stay, is significantly associated with severe morbidity such as multiple organ failure, critical illness myopathy, and acute delirium.[3] Sepsis was the most expensive condition treated in hospitals in the United States in 2013.[4] Sepsis was also the most expensive condition billed to Medicare and Medicaid, with aggregate hospital costs of $14.55 billion (8.2% of national health-care costs) billed to Medicare and $3.35 billion (5.3% of national health care costs) billed to Medicaid in 2013.[4] Along with monetary health care costs of sepsis treatment for the individual, hospitals, government, and the taxpayers, there is a lasting impact on lives of sepsis survivors and their caregivers affecting their health-related quality of life (HRQOL).[5]

Sepsis diagnosis criteria have changed many times due to complexities and lack of accuracy in diagnosis of sepsis and septic shock. The most recent definition, Sepsis-3 criteria, recommended use of the Quick Sequential Organ Failure Assessment (qSOFA) score for early diagnosis of sepsis: (1) respiratory rate (RR) ≥22/min (2) change in mental status, and (3) systolic blood pressure (SBP) ≤100 mm Hg.[6] Sepsis-3 defined sepsis as suspected or documented infection and an acute increase of ≥2 of the SOFA points; and septic shock as subset of sepsis in which the severe acute circulatory and cellular metabolic failure leads to increased mortality.[6] Sepsis-3 also defined septic shock as presence of sepsis and vasopressor therapy needed to elevate mean arterial pressure ≥ 65 mm Hg and lactate >2 mmol/L (18 mg/dL) despite adequate fluid resuscitation.[6]

Early goal-directed therapy (EGDT) for sepsis includes rapid and early recognition of sepsis, early resuscitation if applicable, early antibiotics, and early eradication of the source of infection to improve patient outcomes.[7] [8] [9] Recognition of sepsis in ED was associated with higher compliance to the Surviving Sepsis Campaign 3-hour resuscitation and management bundle leading to decreases in patient mortality.[9] Machine learning (ML) models have been useful in predicting sepsis and in decreasing sepsis mortality rates and 30-day readmission rates for inpatients.[10] [11] Recently, Teng and Wilcox reviewed several models for sepsis prediction, including different ML model structures, feature selection, and data sample size methods, and demonstrated such predictive analytics tool are beneficial in early identification of sepsis and thereby improve patient outcomes.[12] This study moves one step further and suggests potential data elements to be included in such models to improve their prediction/detection performance without introducing any delay in running the models when applied in clinical workflow.

Prehospital data, also referred as emergency medical service (EMS) data or ambulance records, are data elements recorded by the ambulance or prehospital care services. Prehospital data are sources of untapped information that can be valuable in early identification of sepsis and in facilitating clinical decision-making for EGDT in ED. Advancement in technology providing interoperability between prehospital and ED data enable use of prehospital data for health information exchange and clinical decision-making in ED.[13]

Despite several efforts in predicting early detection of sepsis in ED, by using ML approaches, the importance of using prehospital data has not been reviewed yet. Therefore, this integrative review of literature first summarizes the current use of prehospital data in developing ML-based clinical decision models in ED, and then focuses on identifying potential contributing data elements in prediction of early detection of sepsis in ED using ML models. Aims of this review of literature are to (1) summarize the use of prehospital data in developing ED prediction models, (2) review use of ML approaches in sepsis prediction/early identification in ED, and (3) introduce potential prehospital data elements that can be helpful in prediction of sepsis in ML models.

Methods

The literature search strategy included two separate search methods related to use of pre-hospital data in ED and prediction or identification of sepsis in ED using ML models. [Fig. 1] shows the breakdown structure of these studies based on their purposes.

Fig. 1 Search strategy for prehospital data and sepsis in emergency department (ED).

The literature search for the use of prehospital data in ED included English language, peer-reviewed articles published between the years July 2015 and July 2020 in PubMed, the Institute of Electrical and Electronics Engineers (IEEE) Xplore, and Google Scholar. The inclusion criteria for this review included studies that used documented prehospital data prior to arrival to ED for the purpose of decision-making in ED for adult patients (age ≥18 years). Similarly, the exclusion criteria for this search related to use of prehospital data include any nonpeer reviewed articles that did not use prehospital data or data documented prior to arrival to ED. Also included were articles for clinical decision-making in ED for the purpose of ED operations, diagnosis in ED, or treatment in ED. Articles that were not in English language were excluded from the review.

Initial search strategy using key terms [“prehospital data” OR Emergency Service] AND [Emergency Department] AND [sepsis OR septic] AND [machine learning OR artificial intelligence] generated 16 articles. “Prehospital data” were used to reduce the number of nonrelevant articles produced without the quotation marks. After exploring the articles through manual inspection, none of the articles found used prehospital data as variables in ML models for clinical decision-making related to sepsis in ED. Therefore, the search was divided into two different search strategies. One search strategy focused on the use of prehospital data for decision-making in ED. Second search strategy focused on sepsis in ED using ML regardless of the use of prehospital data.

In the search related to the use of prehospital data in ED, the key terms included [“prehospital data” OR “Emergency Medical Service Data”] AND Emergency Department AND [predict OR prediction model OR machine learning OR artificial intelligence]. In total, 200 articles were found between the three databases which were reduced to 113 with filters for within 5 years and peer-reviewed articles. After removal of duplicates from Google Scholar, 80 articles were manually assessed for relevance. Relevance was based on use of prehospital data collected prior to arrival in ED as a data element or variable in clinical decision-making in ED. In total, 12 articles related to use of prehospital data in clinical decision-making in ED were included in this review.

The literature search for the ML models for early identification of sepsis in ED included English language, peer-reviewed articles published between the years 2015 to 2020 in PubMed and IEEE Xplore. Key terms included [sepsis OR Septic] AND Emergency Department AND [prediction OR machine learning OR Artificial Intelligence]. PubMed initially generated 28 articles and IEEE generated 9 articles that included peer-reviewed conference papers. The filters of within 5 years, peer-review, and assessment for duplicates did not change the total numbers of articles. After manually inspecting for relevance, 12 additional studies were included in the review. Criteria for relevance included use of ML or artificial intelligence in predictions related to sepsis using data that are generally collected in ED.

Results

Use of Prehospital Data

Limited prehospital data have been used in developing ED prediction models to improve ED outcomes. Of the 12 studies reviewed that used prehospital data for ED decision-making; 1 study improved ED operations by forecasting number of arrivals to reduce overcrowding in ED[14]; 4 studies predicted patient outcomes such as in-hospital mortality, survival rate, and return of spontaneous circulation (ROSC)[15] [16] [17] [18]; 2 studies identified specific risk or early warning scores for patient outcomes such as higher acuity or short-term in-hospital mortality[19] [20]; 2 studies made decisions in the field prior to arrival to ED such as triage patient disposition to specialized centers with appropriate medical capabilities (e.g., trauma centers or aortic surgery centers)[21] [22]; and 3 studies identified time-sensitive conditions.[21] [22] [23] All studies reported improvement in ED operations or ED outcomes as described in [Table 1].

Table 1 Summary of studies using prehospital data for clinical decision-making in emergency department (ED)

Reference	Model	Target(s)	Target class/distribution	Predictors			Performance/result
				Pre-EDA	Post-EDA	TTP
[13]	Poisson's time-series regression model	No. of arrivals within a forecasting horizon of 1, 2, and 3 hours	Numeric	Arrival notification	Sampling the data on 5-minute intervals; the number of arrivals in the previous 1, 2, and 3 hours	1 hour prior to EDA	Notification prior to EDA contributed to higher forecasting accuracy especially on a 1-hour horizon
[23]	Multivariable logistic regression model and SuperLearner (SL) model	Prediction for acute aortic syndromes (AAS) and performance of undertriage and overtriage	Binary (62.4% positive)	Referred from PC; helicopter Tpt; Htn; smoking; DM; EDA; pain: sudden, abd, thoracic, ripping/tearing, migrating; severe pain intensity; pain scale rate pre-analgesic; focal neuro deficit and type; Htn or use of vasodilatators at EDA; mechanical ventilation at EDA; prehospital CA	EDA date and time; DOB; sex; age; origin of the patient; Hx of abd and thoracic AA; SG: aortic, valve, CABG; connective tissue disease; aortic manipulation; AD; familial AD; Inflammatory vasculitides; pulse deficit; SBP differential; new AI murmur; use of vasoactive Rx; CV collapse; ECG; ST elevation; pericardial effusion; cardiac tamponade; syncope; GCS < 8; GI bleeding; hemoptysis; LI; In-hospital CA; ADDRS; TTE; CT-scan from PC and/or AC; need for SG; Time-to-SG; SAPS II; Final VASC Dx and alternative Dx hospital LOS; status at hospital D/C	24 hours after EDA	SL model with AUC of 0.87 in the training cohort and 0.73 in the validation cohort outperformed logistic regression model with AUC of 0.68 in the training cohort and 0.67 in the validation cohort (p < 0.05)
[18]	The National Early Warning Score (NEWS) logistic regression model	Death within 1 day of emergency medical service dispatch	Binary (1.1% positive)	RR; SpO2; use of supplemental oxygen; temperature; SBP; HR; Level of consciousness	–	prior to arrival to ED	The AUC for primary outcome of death within 1 day was 0.840 (95% CI: 0.823-0.858)
[19]	Composite risk score based on gradient boosting model	Outcome acuity level (transferred to floor, transferred to critical care) or in hospital mortality	Multiclass (1. admissions: 52%; 2. critical care: 5.8%; 3. mortality within 2 days: 1.2%)	Age, gender, DOB, hour and month that the call was received, distance to the nearest ED, and prior contacts with EMD center by the patient, clinical characteristics of the call, times to reach the incident, on scene, and to the hospital, VS, patient Hx, Rx and procedures administered, and the clinical findings of ambulance staff	–	Prior to EDA	ML models based on ambulance data with AUC of 0.79 (0.78–0.80) for hospital admission, AUC of 0.79 (0.77–0.81) for critical care, and AUC of 0.89 (0.87–0.92) for 2-day mortality outperformed NEWS score with AUC of 0.66 (0.65–0.67), 0.75 (0.73–0.77), and 0.85(0.81–0.88), respectively
[22]	Deep learning approach with CNN and LSTM	Shock/no shock decision. detection of VF	Binary	ECG records from out-of- hospital CA	–	<3 seconds	Enabled an accurate shock/ no shock Dx in <3 seconds; decision to shock/ no shock was considered an important prehospital data for ED
[16]	ROTEM-based logistical regression model	ROSC in ED	Binary (31% positive)	POW during CPR; presence of activation; initial rhythm: VF, VT, PEA, asystole	VF, VT, PEA, asystole; time from onset to blood sampling after ED admission; WBC; HgB; platelet count; PT; INR; aPTT; fibrinogen; FDP; D-dimer and lactate; EXTEM; FIBTEM	After ROTEM results in ED	Rotem based predictors of ROSC model had low sensitivity (40.9%) but high specificity (94.7%) and accuracy (75.0%)
[14]	Random forest	30-day survival rate post–out-of-hospital CA	Binary	Initial rhyhm; Age; time- CA to CPR, CA to 911 call, EMS dispatch to arrival; AED; Witness Arrest, AED before ambulance arrival, cause of CA, calendar year, region, time during day, and sex	–	prior to arrival to ED	Initial rhythm; age; early CPR (CPR, time to CPR and CPR before arrival of EMS) were top 3 most important predictors to 30-day survival post out of hospital CA
[17]	Random forest	1-year survival rate post–out-of-hospital CA	Binary (6.35% positive for training set and 4.27% positive for test set)	Medical Hx; ADL before CA; healthy or mild, moderate, and severe disability; PVS; Site; POW; bystander CPR; patient status; cardiac pulmonary, cardiac, and resp arrest; ROSC; presence of SB and pulse; RPD; LPD; PLR; ECG; JCS; CPR; use of CCD; AED; airway securing and method; VA; adrenaline use; removal of airway obstruction; ROSC during Tpt; change of ECG waveform during Tpt; no. of EMT; Time: between EMS perception and contact, EMS perception and CPR, EMS perception and EDA, EMS arrival and contact, EMS contact and CPR, CPR and EDA; other party	–	After EDA	Trained using 35 variables for 1-year survival showed AUC values of 0.943 (95% CI [0.930, 0.955]) and 0.958 (95% CI [0.948, 0.969]), respectively
[24]	Gradient boosting decision tree	Injury severity score (ISS) ≥ 16, linked to lower mortality rate; resource-based outcome measure to define need for specialized trauma care	NA	Age, gender, VS, GCS, SBP, DBP, HR, RR, Intubation, SpO2, Mechanism of injury: MVA, Motorcycle accident, Moped, scooter accident- MVA, pedestrian- MVA, different, gunshot, stab wound, struck with blunt object, fall at same level and at higher level; asphyxia, burns, body surface, injury type, penetrating injury to head, neck, torso, and extremities proximal to elbow and knee, flail chest, paralysis, open or depressed skull fracture	–	Prior to EDA	Aim of the study is to create a new prediction model to attain acceptable undertriage rates and to minimize mortality rates for patients in need of specialized trauma care. Validation and performance results are pending
[21]	Prediction of acute coagulopathy of trauma score	Acute traumatic coagulopathy INR > 1.5	Binary (5.9% positive)	Shock index, age, GCS, mechanism of injury, intubation, CPR, chest decompression	–	Prior to EDA	The prediction of acute coagulopathy of trauma score with AUC of 0.80 (0.72–0.88) outperformed coagulation of severe trauma score with AUC of 0.70 (0.60–0.80; p = 0.032)
[20]	Comparison of logistic regression, random forest, support vector, eXtreme gradient boosting	Presence of large vessel occlusion	Binary (43.3% positive)	Pre EDA (level 1): age; gender; presence of speech deficits; facial weakness; Lt and Rt sided facial weakness; limb weakness; Lt and Rt sided limb weakness	Post-EDA (Level 2): preexisting medical conditions: DM, Htn; current and Hx of smoking; SBP and DBP; GCS; AF Hx, atherosclerosis, cardioembolism, and valvular heart disease; level 3: CT scan results	Level 1 prior to EDA; level 2 after H&P in ED or EMS; Level 3 after CT scan results are read	Comparing all four machine learning algorithms, the eXtreme Gradient Boosting method gave robust and accurate performance in all 3 levels.
[15]	Regression models	Death within 14 days post trauma	Binary (22.8% positive)	Gender; age; level of pupil reactivity at admission; GCS at the site	GCS at EDA; motor component score of the GCS; presence of hypoxia and hypotension; midline shift bigger than 5 mm; brain herniation detected on CT (defined as effacement of the third ventricle or the basal cisterns); hemorrhage: SAH, epidural, subdural and intracerebral	After CT scan results are read in ED	When predicting in-hospital mortality, random forest was the best performing model (AUC = 0.838), closely followed by generalized partial least squares (AUC = 0.831), stochastic gradient boosting (AUC = 0.823), and penalized discriminant analysis (AUC = 0.803)

Abbreviations: AA, aortic aneurysm; Abd, abdominal; AC, aortic center; AD, aortic dissection; ADDRS, aortic dissection detection risk score; ADL, activities of daily living; AED, automatic external defibrillation; AF, atrial fibrillation; AI, aortic insufficiency; aPTT, activated partial thromboplastin time; AUC, area under receiving operating curve; CA, cardiac arrest; CABG, coronary arter bypass graft; CCD, chest compression device; CI, confidence interval; CNN, convolutional neural networks; CPR, cardio-pulmonary resuscitation; CT, computerized tomography; CV, cardiovascular; D/C, discharge; DBP, diastolic blood pressure; DM, diabetes mellitus; DOB, date of birth; Dx, diagnosis; ECG, electrocardiogram; ED, emergency department; EDA, ED arrival time; EMD, emergency medical dispatch; EMS, emergency medical service; EMT, emergency medical technician; EXTEM, extrinsic coagulation pathway; FDP, fibrin degradation products; FIBTEM, function of fibrinogen pathway in the extrinsic pathway; GCS, Glasgow coma scale; GI, gastrointestinal; H&P, history and physical; HgB, hemoglobin; HR, heart rate; Htn, hypertension; Hx, history; INR, International Normalized Ratio; JCS, Japanese coma scale; LI, limb ischemia; LOS, length of stay; LPD, left pupil diameter; LSTM, long short-term memory; Lt, left; MVA, motor vehicle accident; Neuro, neurologic; PC, primary center; PEA, pulseless electric activity; PLR, pupil light reflex; POW, presence of witness; PT, prothrombin time; PVS, persistent vegetative state; Resp, respiratory; ROSC, return of spontaneous circulation; ROTEM, thromboelastometry; RPD, right pupil diameter; RR, respiratory rate; Rt, right; Rx, drugs; SAH, subarachnoid hemorrhage; SAPS II, simplified acute physiology score II; SB, spontaneous breathing; SBP, systolic blood pressure; SG, surgery; Site, site of incidence; SpO₂, oxygen saturation; TPT, transport; TTE, Transthoracic echocardiography; TTP, time to predict (the earliest time that the prediction model can be run based on the predictors); VA, vascular access; VASC, vascular; VF, ventricular fibrillation; VS, vital signs; VT, ventricular tachycardia; WBC, white blood count.

Development and validation of ML models using prehospital data are useful in early identification of time-sensitive conditions, such as trauma, stroke, and shockable rhythms post–out-of-hospital cardiac arrest (CA).[16] [21] [22] For instance, prediction of acute coagulopathy of trauma (ACT) score, based on prehospital data, was developed for early identification of ACT, a complication of trauma that may require early goal-directed treatment of massive transfusion for patient survival.[22] Therefore, the early information that the prehospital data provide can be an asset in improving time to predict time-sensitive conditions in ED.

Machine Learning Models Using Prehospital Data

Of the nine studies included in the review, common prehospital data elements used in ML models targeting early detection of time-sensitive conditions or prediction of patient outcomes include the following: (1) demographics, such as age and gender; chief complaints[15] [16] [20] [21] [22] [24] [25]; (2) prehospital level of consciousness such as Glasgow coma scale (GCS)[19] [20] [22] [25]; (3) prehospital vital signs recorded in the ambulance or in the field[16] [21] [22]; (4) prehospital actions, characterized as intubation, cardiopulmonary resuscitation (CPR), and chest decompression[16] [18] [21] [22]; or (5) prehospital situational assessments, described as mechanism of injury, witnessed arrest or CPR initiation, and presence of speech deficit and facial and limb weakness prior to arrival to ED.[15] [18] [21] [22] [Table 1] summarizes studies using prehospital data for clinical decision-making in ED based on their targets and modeling approaches. Benefits of using prehospital data to achieve ED outcomes were noted in all of the studies included in the review.

All 12 ML articles used prehospital data to impact ED clinical decision-making and ED outcomes. Accordingly, [Table 1] divides the data elements used for each study into pre–emergency department arrival (pre-EDA) and post-EDA and provides further insights about the earliest time that the prediction model can be run based on the predictors (referred in the table as time to predict, or TTP) in each study. Pre-EDA includes data from emergency call centers, dispatches stations and ambulance records collected prior to arrival to ED. Post-EDA includes data collected during hospitalization starting from the arrival to ED.

Use of Machine Learning–Related to Sepsis in ED

Early and accurate prediction of sepsis in ED can improve patient outcomes such as decreased inpatient mortality rate, length of stay, and readmission rates.[26] Advantage of ML models includes the ability to predict patient outcomes hours prior to onset of actual outcome such as diagnosis of sepsis and septic shock.[27] [28] [29] [30] [31] From the 12 studies included in this review, only 1 study used prospective research design in a clinical setting.[26] All other studies used retrospective data for ML modeling. A meta-analysis by Fleuren et al supported that ML models can accurately predict sepsis onset ahead of time in ED, floor, and ICU, providing a novel approach to early identification of sepsis where the biomarkers and screening tools, such as systematic inflammatory response syndrome (SIRS) and SOFA criteria, fail to include all the clinical relevant information.[32] The meta-analysis also showed the impact of prediction hours before sepsis onset on pooled area under receiving operating curve (AUC) for the ML models, further indicating that the prediction hours before onset of sepsis with relatively high AUC through ML model is plausible.[32]

Data Elements in Detection of Sepsis

The ML models used in the ED are evaluated based on their target, modeling approach, data element characteristics, time to predict, and AUC. [Table 2] provides a list of studies found in the search and summarizes them based on their target and modeling approach. It also includes a detailed list of variables used to develop the ML models for each study. Furthermore, the table provides the AUC and TTP which determine the earliest time that all required data elements for running the model are collected and the performance of the model. Therefore, for each target, by considering both TTP and prediction performance, this table helps the reader understand the practicality of the developed models if applied within clinical workflow.

Table 2 Summary of studies with prediction/early identification of sepsis using machine learning with their list of predictors in emergency department (ED)

Reference	Model	Target(s)	Predictors	Performance/result
[36]	Logistic regression with L1 regularization (LASSO)	30-day in-hospital mortality among septic patients	Mean RR; mean HR; NN50; TINN; power norm: very low, low, high frequency; LF/HF, ratio of LF power to HF power; SBP; temp; GCS; age; approximate entropy; sample entropy; detrended fluctuation analysis. TTP: 2 hours after EDA	12 selected HRVTS features outperformed model that only included patient demographics, vital signs and one HRV measures taken at triage. AUC: 0.82
[34]	Random forest, classification and regression tree (CART) model, logistic regression model	28 days in-hospital mortality	SpO2; RR; BP; BUN; albumin; intubation in ED; need for vasopressors; age; RDW; potassium; AST; HR; acuity level (triage), ED impression (Dx), CO2 (laboratory), ECG performed, beta-blocker (home medication), cardiac dysrhythmia (primary medical history). TTP: after ED stay	Machine learning outperformed statistical models. AUC: 0.86
[27]	Support vector model	Dx of an infection	Age; gender; acuity; SBP; DBP; HR; pain scale, RR, SpO2; temperature, free text chief complaint, free text nursing assessment. TTP: after ED stay	Best performing models included free text (NLP). AUC: 0.89
[28]	Gradient tree boosting	Detect onset of sepsis, severe sepsis and septic shock	SBP; DBP; HR; RR; SpO2 and temperature. TTP: at the onset in ED, ICU and floor	Includes ED, ICU and floor. Early detection 4 hours before for septic shock. AUC: 0.92
[33]	Convolutional neural network plus softmax	Predict mortality over 72 hours and 28 days	5 BP; GCS; WBC; HgB; lymphocyte count; PT-INR; BUN; creatinine; bilirubin; AST; ALT; troponin; pH; bicarbonate level; atypical lymphocyte; promyelocyte; metamyelocyte; myelocyte; Sodium ion; potassium ion; albumin; BG; RDW-SD, MCV; RDW-CoV; Base excess; MCH; MCHC; MAP; RR; temperature, HR, Age, Sex, qSOFA score, shock episode, liver cirrhosis, DM; CRF; CHF; CVA; solid tumor, RI, UTI, STI; IAI; other infections, bacteremia, antibiotic used within 24 hours. TTP: 24 hours after EDA	Significant comparisons between different models; outperformed all machine learning and statistical models. AUC: 0.94
[25]	Machine learning algodiagnostic (MLA)	In-hospital mortality	HR, RR, BP, SpO2, temperature. TTP: 1 hour after EDA	Prospective study noted decrease in sepsis-related in-hospital mortality rate, length of stay and 30-day readmission rate 60.24%, 9.55% and 50.14%, respectively. AUC: 0.91
[26]	Multivariate linear regression	Dx of sepsis	Age, HR, RR, temperature, SBP, DBP, MAP, SpO2, HR to SBP ratio. TTP: After EDA with first vital signs.	HR: SBP ratio was higher in patients with sepsis than patients with trauma, stroke and acute coronary syndrome. ML (74%) has higher accuracy than SIRS (34%).
[35]	Multivariate logistic regression, decision tree, and naïve Bayes' classifier	In-hospital mortality	SIRS criteria: temperature, HR, RR, WBC count. qSOFA criteria: SBP, GCS, RR TTP: after EDA with first vital signs.	Low and similar accuracy between the three models with variables related to SIRS and qSOFA. AUC: ranging from 0.622 to 0.696
[30]	SVM, gradient boosting machine with Bernoulli's loss, random forest, multivariate adaptive regression splines, lasso and ridge regression	Septic shock within 24 hours of arrival	Sex; chief complaints; initial vital signs: SBP, DBP, HR, RR, SpO2 and temperature; initial level of consciousness- AVPU scale; WBC, differential counts; RDW; platelet count; PT; INR; fibrinogen; BUN; sodium; potassium; chloride, creatinine, AST, ALT, ALP, total bilirubin; albumin; and C-reactive protein. TTP: 24 hours after arrival	All machine learning classifiers significantly outperformed statistically calculated models. Embedding chief complaints was statistically significant in improving performance AUC: ranging from 0.883 to 0.929.
[37]	eXtreme gradient boosting, light gradient boosting and random forest	3-day mortality among patients with suspected infection in the ED	The qSOFA criteria, including systolic blood pressure, respiratory rate, and mental status. TTP: After EDA with first vital signs	The machine learning models outperformed qSOFA score using the same variables in predicting 3-day mortality with suspected infections in ED AUC: model 0.86.
[32]	Gradient-boosted tree models	Detection of sepsis	LA, shock index; WBS; neutrophils, change in LA; BG; BUN; RR; albumin; SBP; serum creatinine; temperature. TTP: after EDA after laboratories are resulted	Models' ROS was more sensitive and precise compared with qSOFA score at 1 hour and at 24 hours. AUC (95% CI): 0.93–0.97
[29]	Decision tree, discriminant analysis, logistic regression, KNN, neighbors, ensemble classification, SVM, and neural network	Identification of sepsis	Age, dialysis, mobility, mentation, SBP, HR, RR, LA, temperature, and WBC. TTP: first 6 hours EDA	Although the fine Gaussian SVM showed highest sensitivity (95.8%), NN model had the highest accuracy (92.08%), high sensitivity (92.33), and high specificity (92.33%) AUC: 0.92

Abbreviations: ALP, alkaline phosphatase; ALT, alanine aminotransferase; AST, aspartate aminotransferase; AUC, area under receiving operating curve [with 95% confidence interval]; AVPU scale, alert, verbal, pain and unresponsive scale; BG, blood sugar; BP, blood pressure; BUN, blood urea nitrogen; CHF, congestive heart failure; CoV, coefficient of variation; CRF, chronic renal failure; CVA, cerebral vascular accident; DBP, diastolic blood pressure; DM, diabetes mellitus; Dx, diagnosis; Dx, diagnosis; ECG, electrocardiogram; EDA, ED arrival time; GCS, Glasgow coma scale; HgB, hemoglobin; HR, heart rate; IAI, intra—abdominal infection; ICU, intensive care unit; INR, International Normalized Ratio; KNN, k-nearest neighbors; LA, lactic acid; MAP, mean arterial pressure; MCH, mean corpuscular hemoglobin; MCHC, mean corpuscular hemoglobin concentration; MCV, mean corpuscular volume; NN50, RR number of consecutive RR triangular index (TINN); pH, power of hydrogen (pH); PT, prothrombin time; qSOFA, quick sequential organ failure assessment; RDW, red cell distribution width; RI, respiratory infection; RN, registered Nurse; ROS, development of risk for sepsis; RR, respiratory rate; RT, respiratory therapist; SBP, systolic blood pressure; SD, standard deviation; SpO2, oxygen saturation; STI, soft tissue infection; SVM, support vector machine; TINN, baseline width of a triangle fit into the RR interval histogram using a least squares; TTP, time to predict (The earliest time that the prediction model can be run based on the predictors); UTI, urinary tract infection; WBC, white blood count.

Although ML models have been used to detect early infection, sepsis, and septic shock, research related to early identification of sepsis or septic shock in patients, specifically on arrival to ED is still limited.[27] [28] [29] [30] [31] [33] Common variables identified from the 12 articles in the review for detection of infection, sepsis, or septic shock in all settings, such as ED, intensive care units, and floors, include SBP, diastolic blood pressure (DBP), heart rate, RR, peripheral capillary oxygen saturation, temperature, and chief complaints,[27] [28] [29] [31] many of which often are not available in electronic health record (EHR) on patient arrival to ED due to lack of interoperability. For instance, Mao et al developed a gradient tree boosting model using data from only six vital signs: SBP, DBP, heart rate, RR, peripheral capillary oxygen saturation, and temperature.[29] This model was able to predict sepsis at the onset with high AUC (0.92) and septic shock 4 hours in advance with high AUC (0.96).[29] The model was also able to predict severe sepsis 4 hours in advance with higher AUC (0.85) than the onset time for statistically calculated SIRS AUC (0.75).[29] This study used data from ED, critical care and regular floors and required 3 hours of measurements of all six vital signs. Model performance might be different if limited to data collected during the early ED visit which may not have continuous vital signs recorded for three hours. Prehospital data can provide an opportunity to use a predictive model on patient arrival that includes these additional data elements.

Only four studies used ML models to predict early onset of infection, sepsis, and septic shock in patients in the ED.[28] [30] [31] [33] In these studies, time needed for prediction of infection or sepsis ranged from 1 hour from the first vital signs on admission to ED to 24 hours after admission to ED.[28] [30] [31] [33] Horng et al found that diagnosis of infection in ED could be predicted more accurately by using free-text chief complaints and nursing assessments along with structured data, such as vital signs and demographics.[28] This study used a variation of NLP in ED to uncover latent data from chief complaints and nursing assessment notes for early detection of infection. Mohamed et al used neurological assessment in the form of mentation along with other common variables to identify sepsis through multiple models where a neural network model performed the best with greater than 90% accuracy.[30] Due to limitation in available data, different tools, such as NLP, and uncommonly used data elements, such as neurological assessment in the form of mentation, may be needed for early prediction of sepsis.

From the 12 studies, 10 defined their diverse methods of handling missing values. Two studies included forward filling imputation[26] [29]; two studies imputed by means, modes or median[31] [34]; two studies converted missing values to categorical or nominal values[30] [35]; one study used multivariate imputation by chained equations method[36]; one study used linear interpolation method[37]; one study excluded patients with missing data[38]; and one study automatically imputed physiologically normal values in the missing vital signs data.[28] The diversity in handling of the missing values especially with vital signs raises concerns related to functionality and ethics of ML model in clinical settings.

Discussion

The purpose of this integrative literature review was to discuss the importance of utilizing prehospital data elements in ED, summarize their current use in developing ML-based prediction models, and specifically identify those data elements that can potentially contribute to early identification of sepsis in ED when used in ML-based approaches. Prehospital data have been instrumental in the development and validation of statistical and ML models with the purpose of improving early detection of time-sensitive patient conditions, such as trauma, stroke, and shockable cardiac rhythms post–out-of-hospital cardiac arrest; and patient outcomes such as inpatient mortality, survival rate, and ROSC.[16] [21] [22] Currently, limited studies are available related to detection and prediction of sepsis in ED, and there was no study found related to detection of sepsis in ED using prehospital data. This gap in knowledge may be due to limitations in availability of prehospital data in the EHR and difficulty in diagnosis of sepsis in ED. There is an opportunity for researchers to develop and evaluate prediction models for early detection of sepsis using prehospital data in the ED in the future.

ML has been found to be superior in performance in terms of accuracy, specificity, and sensitivity compared with statistically calculated screening tools for sepsis prediction.[16] [20] [21] [24] [32] [38] Based on current and previous definitions of sepsis, the early detection of sepsis has been linked with the following screening tools: SIRS, the modified Early Warning Scores (MEWS), and SOFA and qSOFA scores.[6] [39] However, SIRS, MEWS, SOFA, and qSOFA scores when compared with ML models have performed poorly in identification of sepsis and in predicting inpatient mortality from sepsis in ED.[27] [33] [35] [38] In a meta-analysis study by Islam et al, ML models showed higher accuracy for sepsis detection as evidenced by the AUC of 0.89 compared with sepsis screening tools such as SIRS, MEWS, and SOFA score with AUC of 0.70, 0.50, and 0.78, respectively.[39] Contrarily, two sets of decision tree models using the same variables as qSOFA and SIRS generated similar and low AUC and sensitivity.[36] Therefore, although the ML models in general outperform statistically calculated sepsis screening tools, the performance of the prediction models depend highly on the selection of the set of variables or data elements used to create the model.

As described in [Table 1], use of prehospital data could help expedite certain predictions in ED if the required data elements were collected earlier during prehospital care. For instance, a promising use of ML in ED using prehospital data could reduce overcrowding by managing the availability of personnel.[14] However, use of prehospital data are limited by the quality and availability of the data. Mashoufi et al conducted a survey among three groups of EMS stakeholders: data producers, data collectors, and data consumers.[40] They concluded that the quality of EMS data with respect to their usability, completeness, and compatibility are still low.[40] An additional caveat for developing an accurate prediction/early detection model in ED is balancing TTP, and the helpfulness of the inputted variables. TTP will be determined by the time that all data elements become available for analysis. For instance, Duceau et al used 32 prehospital and postarrival variables that included electrocardiogram (ECG), computed tomography (CT) scan, and transthoracic echocardiogram to predict acute aortic syndromes (AAS) and assess the performance of undertriage and overtriage.[24] Use of data elements that take a longer time to result such as diagnostic tests provides accuracy in performance but may delay the time to predict.[41] Contrarily, only using insufficient prehospital data, one model may be able to deliver prediction at arrival time but with potential cost of reduced performance. Therefore, balancing performance and time to predict is necessary in clinical setting with consideration to quality and availability of data.

This literature review identifies data elements that may be used for early prediction of sepsis in ED using ML models. ML models using prehospital data elements have been applied in ED, although not for sepsis, and have reported high performance in improving ED operations and predicting ED outcomes. Data elements found helpful in early prediction or detection of sepsis in ED are also highlighted, although none were from prehospital data. Therefore, the prehospital data elements, identified in [Table 2], and the data elements specific to sepsis, identified in [Table 2], may be combined to improve early identification of sepsis in ED. This integrative literature review emphasizes the current gap in use of prehospital data and its potential in supporting early identification of sepsis in ED using ML and provides data elements that may facilitate further predictive analysis research.

Limitations

Some limitations should be noted. One limitation is related to the literature search itself and its scope. The search for this literature review focused on the use of prehospital data for ED decision-making and the ML models for sepsis in ED. Any other studies, including interventional studies, not related to the two search strategies and our search terms were not included. The limited search terms used in this review excluded some of the studies that have been included in other reviews such as meta-analysis by Fleuren et al and a systematic review by Kareemi et al.[32] [42] The purpose of our review also differed from these systematic reviews as the results focused on studies that used prehospital data for ED decision-making.

Another limitation is the use of retrospective research designs for most of the studies included in this review and lack of implementation of ML models in a clinical setting. Only one prospective study related to sepsis by McCoy and Das found reduction in in-hospital mortality rate by 60.24% postimplementation of sepsis prediction score alert system on floors, ICU, and ED.[26] That study also observed improvement in patient outcomes: a decrease in sepsis-related hospital length of stay by 9.55%, a decrease in 30-day readmission rate by 50.14%, and an increase in 3-hour sepsis bundle compliance by 72.7%.[26] However, the impact on patients specifically in ED and accuracy of the sepsis prediction score in ED was not analyzed in the study. Therefore, more research with the target outcome of early prediction or detection of sepsis focusing on ED patients using prehospital data are necessary to impact real-time clinical decision-making in ED and to improve patient outcomes.

There are also limitations in the use of prehospital data in practice, such as ML models using prehospital data that only benefit patients who arrive by ambulance. The lack of generalizability to all patients who arrive to ED limit the scope of the use of prehospital data. Limited access to data elements further limits the use of prehospital data. Despite the progress in recent years,[43] [44] there is still a lack of interoperability that may limit the number and types of the prehospital data available for ML models.[45] Moreover, assuming the health care system has interoperability with EMS data with real-time access to the recorded data, availability of the included data elements at the time of prediction is crucial in designing clinical decision-support systems. Martin et al identified interoperability, accurate match algorithms, security, and wireless connectivity as potential barriers to adoption of prehospital health information exchanged.[13] Further development in ED–EMS interface is needed to enable prominent use of prehospital data in ED decision-making and promote research in use of prehospital data for improved patient outcomes. Additionally, incomplete or missing data are often a problem with prehospital data. Using various imputing techniques would likely be fraught with measurement error which would lead to inaccurate models. More research is needed in this area.

Conclusion

Sepsis has a significant impact on patient outcomes and HRQOL and financially impacts patients, their families, hospitals, community, and national health care costs. Sepsis is a time-sensitive condition that requires early detection and EGDT to improve patient mortality and related patient outcomes. Prehospital data in partnership with NLP and ML models can potentially improve clinical decision-making for sepsis in ED. Accessibility and limitations of the prehospital data, appropriateness of the variable selection, and the time it takes to generate a prediction have significant impacts on the performance of the ML models to improve patient outcomes for time-sensitive conditions, such as sepsis in ED.

Although ML models outperform sepsis severity scores and have the potential to predict at the onset or prior to sepsis and septic shock, the performance of the model depends heavily on selection of the variables or data elements. Future implications suggest development, validation, and application of the ML combined with NLP models using prehospital data in clinical settings to identify sepsis earlier and to support related clinical decision making in ED to improve patient outcomes.

Clinical Relevance Statement

With increases in availability of data, systematic use of data are necessary to improve patient care and patient outcomes. With advancements in technology and the right clinical decision-support tools, clinicians have the ability to recognize time-sensitive conditions earlier and thereby improve patient outcomes. ML models that integrate data elements of early identification of sepsis in ED using prehospital data advance the clinical practice by early recognition of sepsis leading to prioritization of patient condition, early intervention, and improved patient outcomes.

Multiple Choice Questions

1. Machine learning models using prehospital data have been used to early identify complications related to all the following time-sensitive conditions except:

   • trauma

   • stroke

   • sepsis

   • out-of-hospital cardiac arrest

   Correct Answer: The correct answer is c. We discussed various machine learning models using prehospital data that have been used in ED for early detection of time-sensitive conditions. To the best of our knowledge, there is no research available that uses prehospital data to early identify sepsis in ED. This gap in current knowledge highlights the need for further research using prehospital data to early identify sepsis in ED.

2. All of the following data elements have previously been used in machine learning models to early identify infection, sepsis, or septic shock in ED except:

   • vital signs in ED

   • free-text chief complaints

   • neurological assessment in form of mentation

   • prehospital vital signs

   Correct Answer: The correct answer is d. Only four studies found focused on early identification of infection, sepsis, or septic shock in ED. Vital signs in ED were the most common data elements used in the studies. Of the four studies, different studies found increase in accuracy in the machine learning models by adding different data elements such as, neurological assessment in the form of mentation, free text chief complaints, and nursing assessment notes.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This literature review did not require IRB approval.

• References

• 1 National Institute of Health. Sepsis. Accessed November 11, 2020 at: https://www.nigms.nih.gov/education/fact-sheets/Pages/sepsis.aspx
• 2 Centers for Disease Control and Prevention. Clinical information. Accessed November 11, 2020 at: https://www.cdc.gov/sepsis/clinicaltools/index.html
• 3 Brück E, Schandl A, Bottai M, Sackey P. The impact of sepsis, delirium, and psychological distress on self-rated cognitive function in ICU survivors-a prospective cohort study. J Intensive Care 2018; 6: 2
• 4 Torio CM, Moore BJ. National Inpatient hospital costs: the most expensive conditions by payer, 2013. Statistical brief #204. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2016
• 5 Gallop KH, Kerr CE, Nixon A, Verdian L, Barney JB, Beale RJ. A qualitative investigation of patients' and caregivers' experiences of severe sepsis*. Crit Care Med 2015; 43 (02) 296-307
• 6 Shankar-Hari M, Phillips GS, Levy ML. et al; Sepsis Definitions Task Force. Developing a new definition and assessing new clinical criteria for septic shock: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016; 315 (08) 775-787
• 7 Levy MM, Rhodes A, Phillips GS. et al. Surviving sepsis campaign: association between performance metrics and outcomes in a 7.5-year study. Intensive Care Med 2014; 40 (11) 1623-1633
• 8 Liu VX, Fielding-Singh V, Greene JD. et al. The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med 2017; 196 (07) 856-863
• 9 Westphal GA, Koenig Á, Caldeira Filho M. et al. Reduced mortality after the implementation of a protocol for the early detection of severe sepsis. J Crit Care 2011; 26 (01) 76-81
• 10 Amland RC, Hahn-Cover KE. Clinical decision support for early recognition of sepsis. Am J Med Qual 2019; 34 (05) 494-501
• 11 Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc 2017; 24 (01) 88-95
• 12 Teng AK, Wilcox AB. A review of predictive analytics solutions for sepsis patients. Appl Clin Inform 2020; 11 (03) 387-398
• 13 Martin TJ, Ranney ML, Dorroh J, Asselin N, Sarkar IN. Health information exchange in emergency medical services. Appl Clin Inform 2018; 9 (04) 884-891
• 14 Asheim A, Bache-Wiig Bjørnsen LP, Næss-Pleym LE, Uleberg O, Dale J, Nilsen SM. Real-time forecasting of emergency department arrivals using prehospital data. BMC Emerg Med 2019; 19 (01) 42
• 15 Al-Dury N, Ravn-Fischer A, Hollenberg J. et al. Identifying the relative importance of predictors of survival in out of hospital cardiac arrest: a machine learning study. Scand J Trauma Resusc Emerg Med 2020; 28 (01) 60
• 16 Amorim RL, Oliveira LM, Malbouisson LM. et al. Prediction of early TBI mortality using a machine learning approach in a LMIC population. Front Neurol 2020; 10: 1366
• 17 Koami H, Sakamoto Y, Sakurai R. et al. Thromboelastometric analysis of the risk factors for return of spontaneous circulation in adult patients with out-of-hospital cardiac arrest. PLoS One 2017; 12 (04) e0175257
• 18 Seki T, Tamura T, Suzuki M. SOS-KANTO 2012 Study Group. Outcome prediction of out-of-hospital cardiac arrest with presumed cardiac aetiology using an advanced machine learning technique. Resuscitation 2019; 141: 128-135
• 19 Pirneskoski J, Kuisma M, Olkkola KT, Nurmi J. Prehospital national early warning score predicts early mortality. Acta Anaesthesiol Scand 2019; 63 (05) 676-683
• 20 Spangler D, Hermansson T, Smekal D, Blomberg H. A validation of machine learning-based risk scores in the prehospital setting. PLoS One 2019; 14 (12) e0226518
• 21 You J, Tsang ACO, Yu PLH. et al. Automated hierarchy evaluation system of large vessel occlusion in acute ischemia stroke. Front Neuroinform 2020; 14: 13
• 22 Peltan ID, Rowhani-Rahbar A, Vande Vusse LK. et al. Development and validation of a prehospital prediction model for acute traumatic coagulopathy. Crit Care 2016; 20 (01) 371
• 23 Picon A, Irusta U, Álvarez-Gila A. et al. Mixed convolutional and long short-term memory network for the detection of lethal ventricular arrhythmia. PLoS One 2019; 14 (05) e0216756
• 24 Duceau B, Alsac JM, Bellenfant F. et al. Prehospital triage of acute aortic syndrome using a machine learning algorithm. Br J Surg 2020; 107 (08) 995-1003
• 25 van der Sluijs R, Debray TPA, Poeze M, Leenen LPH, van Heijl M. Development and validation of a novel prediction model to identify patients in need of specialized trauma care during field triage: design and rationale of the GOAT study. Diagn Progn Res 2019; 3: 12
• 26 McCoy A, Das R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual 2017; 6 (02) e000158
• 27 Danner OK, Hendren S, Santiago E, Nye B, Abraham P. Physiologically-based, predictive analytics using the heart-rate-to-Systolic-Ratio significantly improves the timeliness and accuracy of sepsis prediction compared to SIRS. Am J Surg 2017; 213 (04) 617-621
• 28 Horng S, Sontag DA, Halpern Y, Jernite Y, Shapiro NI, Nathanson LA. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS One 2017; 12 (04) e0174708
• 29 Mao Q, Jay M, Hoffman JL. et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 2018; 8 (01) e017833
• 30 Mohamed A, Ying H, Sherwin R. Electronic-medical-record-based identification of sepsis patients in emergency department: a machine learning perspective. In: International Conference on Contemporary Computing and Applications; February 5–7, 2020, 2020; Lucknow, India
• 31 Kim J, Chang H, Kim D, Jang DH, Park I, Kim K. Machine learning for prediction of septic shock at initial triage in emergency department. J Crit Care 2020; 55: 163-170
• 32 Fleuren LM, Klausch TLT, Zwager CL. et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med 2020; 46 (03) 383-400
• 33 Delahanty RJ, Kaufman D, Jones SS. Development and evaluation of an automated machine learning algorithm for in-hospital mortality risk adjustment among critical care patients. Crit Care Med 2018; 46 (06) e481-e488
• 34 Perng JW, Kao IH, Kung CT, Hung SC, Lai YH, Su CM. Mortality prediction of septic patients in the emergency department based on machine learning. J Clin Med 2019; 8 (11) E1906
• 35 Taylor RA, Pare JR, Venkatesh AK. et al. Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach. Acad Emerg Med 2016; 23 (03) 269-278
• 36 Gupta A, Liu T, Shepherd S, Paiva W. Using statistical and machine learning methods to evaluate the prognostic accuracy of SIRS and qSOFA. Healthc Inform Res 2018; 24 (02) 139-147
• 37 Chiew CJ, Wang H, Ong MEH. et al. Serial heart rate variability measures for risk prediction of septic patients in the emergency department. AMIA Annu Symp Proc 2020; 2019: 285-294
• 38 Kwon YS, Baek MS. Development and validation of a quick sepsis-related organ failure assessment-based machine-learning model for mortality prediction in patients with suspected infection in the emergency department. J Clin Med 2020; 9 (03) E875
• 39 Islam MM, Nasrin T, Walther BA, Wu CC, Yang HC, Li YC. Prediction of sepsis patients using machine learning approach: a meta-analysis. Comput Methods Programs Biomed 2019; 170: 1-9
• 40 Mashoufi M, Ayatollahi H, Khorasani-Zavareh D. A review of data quality assessment in emergency medical services. Open Med Inform J 2018; 12: 19-32
• 41 Bolodeoku J, Ogbeiwi O, Ma K, Sa A. Laboratory tests turnaround time in outpatient and emergency patients in nigeria: results of a physician survey on point of care testing. Int J Med Res Health Sci 2017; 6 (05) 76-81
• 42 Kareemi H, Vaillancourt C, Rosenberg H, Fournier K, Yadav K. Machine learning versus usual care for diagnostic and prognostic prediction in the emergency department: a systematic review. Acad Emerg Med 2021; 28 (02) 184-196
• 43 DePalo P. Electronic health record interoperability across transport medicine [dissertation]. Towson, Maryland: Towson University Institutional Repository; 2014
• 44 Xia Y, Zhou Y, Liu X, Zhao N, Xiao S. How the integration of emergency medical services with Health Information Systems enhances quality of service. Accessed January 5, 2021 at: https://eapj.org/wp-content/uploads/2021/02/Integration-of-Emergency-Medical-Services-with-Health-Information-Systems.pdf
• 45 Nakayama M, Inoue R, Miyata S, Shimizu H. Health information exchange between specialists and general practitioners benefits rural patients. Appl Clin Inform 2021; 12 (03) 564-572

Publication History

Received: 05 August 2021

Accepted: 20 December 2021

Article published online:
02 February 2022