Automating a Manual Sepsis Screening Tool in a Pediatric Emergency Department

Abstract

Objective This article describes the method of integrating a manual pediatric emergency department sepsis screening process into the electronic health record that leverages existing clinical documentation and keeps providers in their current, routine clinical workflows.

Methods Criteria in the manual pediatric emergency department sepsis screening tool were mapped to standard documentation routinely entered in the electronic health record. Data elements were extracted and scored from the medical history, medication record, vital signs, and physical assessments. Scores that met a predefined sepsis risk threshold triggered interruptive system alerts which notified emergency department staff to perform sepsis huddles and consider appropriate interventions. Statistical comparison of the new electronic tool to the manual process was completed by a two-tail paired t-test.

Results The performance of the pediatric electronic sepsis screening tool was evaluated by comparing flowsheet row documentation of the manual, sepsis alert process against the interruptive system alert instance of the electronic sepsis screening tool. In an 8-week testing period, the automated pediatric electronic sepsis screening tool identified 100% of patients flagged by the manual process (n = 29), on average, 68 minutes earlier.

Conclusion Integrating a manual sepsis screening tool into the electronic health record automated identification of pediatric sepsis screening in a busy emergency department. The electronic sepsis screening tool is as accurate as a manual process and would alert bedside clinicians significantly earlier in the emergency department course. Deployment of this electronic tool has the capability to improve timely sepsis detection and management of patients at risk for sepsis without requiring additional documentation by providers.

Keywords

EHR workflow design - emergency medicine - screening tool - pediatric sepsis

Background and Significance

Severe sepsis and septic shock continue to cause significant morbidity and mortality in children worldwide.[1] To that end, the World Health Organization, along with its decision-making body the World Health Assembly, adopted a resolution on May 24, 2017, to improve the prevention, diagnosis, and management of sepsis—a crucial step in reducing the global burden of sepsis.[2] The prevalence of pediatric severe sepsis and septic shock has been increasing; however, advances are being made, and mortality is decreasing.[3] Recent estimates suggest that there are approximately 4,500 children who die every year from sepsis in the United States alone.[4] Delayed antimicrobial therapy beyond 3 hours from sepsis recognition is an independent risk factor for mortality and prolonged organ failure in pediatric sepsis and septic shock.[5] The most recent clinical practice parameters published by the American College of Critical Care Medicine emphasize early, goal-directed therapy within the first hour of identification,[6] as early detection leads to improved patient outcomes through earlier fluid resuscitation and antibiotic therapy.[7] [8] Implementation of sepsis screening tools and bundles to guide treatment has been shown to decrease time to fluid administration and delivery of antibiotics.[9] [10] [11] [12]

Many hospital systems now employ an electronic health record (EHR) and it follows that implementation of an electronic sepsis screening tool should be integrated into patient care. Due to age-specific parameters and the pediatric patient's unique ability to compensate with relatively stable vital signs, automated EHR sepsis screening is not as straightforward as in the adult population. Therefore, automated EHR sepsis screening tools are not readily available for pediatric populations necessitating the use of manual or paper processes in many institutions. Recently, a large children's hospital published a prospective application and study of an emergency department (ED)-based electronic alert to improve recognition of pediatric sepsis. The introduction of a vital sign-based electronic sepsis screening tool with bedside clinician documentation did improve the recognition of sepsis in their pediatric population.[13] Despite the improvement reported in their study, the EHR-based tool relied on a two-stage alerting system that required additional clinician documentation beyond that necessary for routine patient care.

Our institution is a large, urban, freestanding, academic quaternary hospital with nearly 90,000 emergency visits a year. In 2013, our ED started a multidisciplinary sepsis team that initially focused on timely goal-directed therapy for provider-identified patients with severe sepsis or septic shock, known as a “sepsis alert.” To allow tracking of an electronic time zero for sepsis alert activation, a flowsheet row was built into the EHR (Epic Systems, Corp.) in 2014. In 2015, it was determined that a more standardized approach to sepsis screening was the next step and the ED sepsis team adopted the most commonly used septic shock trigger/identification tool ([Fig. 1]) published by the American Academy of Pediatrics Pediatric Sepsis Collaborative.[14] The effectiveness of our interventions were assessed using manual chart review of patients who had been made a sepsis alert by providers in the ED as well as patients with admission to the intensive care unit with severe sepsis or septic shock and those with discharge International Classification of Diseases 10th Revision codes consistent with severe sepsis or septic shock.[15]

Fig. 1 Septic shock trigger/identification tool.

This trigger tool used a combination of high-risk conditions (defined as malignancy, asplenia including sickle cell disease, bone marrow transplant, central or indwelling line/catheter, solid organ transplant, severe mental retardation/cerebral palsy [limited verbal capacity or nonverbal and partial or complete assistance required for all activities of daily living], and immunodeficiency/immunocompromised or immunosuppression) along with eight clinical criteria: abnormalities in temperature, capillary refill, pulse and/or skin exam, mental status as well as hypotension, tachycardia, and/or tachypnea. Patients were considered septic if they met 3 of the 8 clinical criteria or had a high risk condition and met 2 of the 8 criteria. After deployment of this tool, many children were erroneously identified as at risk for severe sepsis when in fact they were not clinically concerning for severe sepsis. Subsequently, the tool was modified to decrease the number of patients erroneously identified as septic. These changes included increasing the respiratory rate for children > 13 to 24 breaths per minute, removing mental status abnormalities due to high user variability, and increased the number of criteria needed to meet the sepsis alert threshold to requiring 4 of 8 clinical criteria or a high-risk condition and 3 of 8 clinical criteria. The paper tool was utilized for 2 years in the ED. The next step for the sepsis team was to integrate the manual sepsis screening tool into the EHR.

Objective

The goal of the sepsis team was to integrate the existing sepsis screening tool into the EHR. The team's goal was to leverage existing documentation in the EHR without requiring additional documentation by providers. In addition, the tool would result in automated notification to providers when the criteria threshold of the tool was met. Here, we will describe the strategy and design method of integrating a manual sepsis screening process into the EHR and share our results.

Methods

To design an electronic version of the manual screening tool, we brought together a multidisciplinary team comprised of ED physicians, nurses, clinical informaticists, and data analysts. With the predominate goal of keeping providers in their routine clinical workflows without requiring additional documentation, the electronic tool needed to leverage standard documentation in the EHR compiled by various provider roles. We developed a process flow map to establish the patterns of clinical tasks and data collection that occur during the ED patient encounter. Through guided conversation, our team “walked through” the patient care continuum for which all arrival patterns, clinical examinations, documentation templates, etc. were reviewed and investigated. The analysis process identified when (e.g., triage, initial assessment, reassessments, etc.), where (past medical history, home medication history, etc.), and by whom, each criterion of the paper trigger tool was obtained and documented. Some items to be evaluated by the electronic tool would be historical within the patient's EHR and not documented in the current encounter such as a high-risk condition.

To determine where a criterion is documented, we created a crosswalk between the paper trigger tool and current, standard documentation fields in the EHR. For example, abnormal temperature is discrete and documented as a numeric value in the vital signs temperature field. After the team completed the crosswalk analysis, all criteria in the paper trigger tool were found as discrete fields in the current EHR with the exception of historical temperature abnormality or provider concern for infection. To ensure appropriate inclusion into the screening tool, a new assessment field was introduced into the intake nursing workflow which mimics the first question of the paper trigger tool: “Does this patient present with concern for INFECTION and/or TEMPERATURE abnormality in the last 4 hours?” ([Fig. 2]). We opted to implement this assessment row in the EHR prior to the electronic sepsis screening tool launch. Compliance with documentation of the assessment was monitored and feedback was provided to nursing stakeholders in the weeks leading up to the electronic screening tool deployment. While the possibility of bias may exist as a result of the introduction of a new assessment field, this question was already present in the paper screening tool. This information is a required element of sepsis screening and needed to be included in the electronic tool that was not identified during the crosswalk analysis.

Fig. 2 Nursing documentation assessment field.

Once satisfied that all the criteria of the paper trigger tool were available in the EHR, we shifted focus to the timing of EHR documentation. Clinical workflows were mapped with further emphasis on when and by whom criteria would be documented. For example, nurses document a capillary refill assessment during the triage exam and again at reassessment. We discovered each criterion would likely be obtained and documented repeatedly, often at routine intervals; however, vital signs and skin and perfusion exams were documented with increased frequency in acutely ill patients and those requiring additional interventions. Essentially, higher acuity patients are likely to have many of the sepsis criteria documented more frequently, thus allowing for a more dynamic screening tool.

We validated the performance of the automated electronic sepsis screening tool through retrospective, EHR data abstraction. We used QlikView, a business intelligence software provided by Qlik (Version 11.2 SR 12. Release Date 6/9/2015. Radnor, Pennsylvania, United States), to evaluate the presence of documented sepsis criteria for all ED encounters at our institution over the 15-month period from January 1, 2016 to March 31, 2017. We focused on ensuring all historical patients documented as a sepsis alert would have been captured by the electronic sepsis screening tool. By evaluating documented criteria every 15 minutes throughout an encounter, we were not only able to ascertain whether a patient would or would not have been captured by the electronic sepsis screening tool but know the approximate time the screening threshold was met.

Once we validated that our criteria could be successfully identified within the EHR, the screening tool was configured using a series of rules or conditions to be evaluated in a scoring logic. Each criterion, such as assessment abnormalities or historical documentation of malignancy, was assigned a score. The presence of criteria documentation is then scored within the screening tool. The EHR continuously evaluates each criterion score until a threshold (positive screen) score is met. The electronic sepsis screening tool evaluates all patients that arrive to the ED and throughout the entire ED encounter. Data elements were extracted and scored from the medical history, medication record, vital signs, and physical assessment.

Once the automatic trigger tool was designed, the team sought to automate provider notification of patients who met sepsis screening threshold. Ultimately, it was determined that an interruptive system alert would be used to notify providers when a patient met the screening threshold which would prompt a sepsis huddle. The tool would include two interruptive system alerts, one to nursing and one to physicians and nurse practitioners. The alert to nursing provides clinical decision support to initiate a sepsis huddle, whereas the physician and nurse practitioner alert displays a list of clinical criteria met and potential management options. The sepsis huddle is comprised of a nurse and a fellow or attending physician or nurse practitioner who would examine the patient at bedside and document the outcome as an intervention of the interruptive system alert. Subsequent management tools such as standard treatment order sets would be hosted in the electronic workflow.

To test the performance of the electronic sepsis screening tool and timing of provider notification, the electronic sepsis screening tool, but not the alerts, was deployed in the live EHR. The provider alerts were not displayed to end users during the 8-week testing period and the manual, paper process remained unchanged. Data were abstracted (sepsis screening scores and timing of alerts) from the patient's EHR behind the scenes from the live environment to assess the effectiveness of the electronic tool. Documentation of a manual screening sepsis alert or time zero (the current, paper process) was subsequently compared with the time when the electronic screening tool would have initiated an interruptive system alert.

Statistical comparison of the electronic sepsis screening tool to the manual process was completed by a two-tail paired t-test. Statistical analysis was performed by R (R: A language and environment for statistical computing. R Core Team, Vienna, Austria, 2017) using Stargazer (Stargazer: Well-formatted regression and summary statistics tables. R Package 5.2, Harvard University, Cambridge, Massachusetts, United States, 2015).

Results

In an 8-week testing period, the electronic sepsis screening tool identified 100% of patients flagged by the manual process (n = 29), and did so, on average, 68 minutes earlier. This difference was found to be statistically significant (p < 0.001, d = 1.52, 95% confidence interval [0.40, 0.84]) ([Table 1]). [Table 2] demonstrates the characteristics of the 29 patients identified as sepsis alerts in the ED during the 8-week testing period. Sixteen were female (55%) and 13 were male (45%). Thirteen patients fell between 12 and 18 years of age, with 1 infant (0–3 months) identified, 4 toddlers (1–4 years), 6 school age children (4–12 years), and 5 adults (> 18 years).

Table 1 Comparison of manual and electronic sepsis screening tools

	Manual documentation	BPA
Alerts	29	29
Mean, min to alert	127	59
Range, min to alert	17–587	1–524
Median, min to alert	89	15

Abbreviations: BPA, best practice advisory; CI, confidence interval.

Note: Paired t-test on log10 of minutes to alert: t(28) = 5.78;p < 0.001; d = 1.52; 95% CI (0.40, 0.84).

Table 2 Sepsis alert patient characteristics (n = 29)

Sepsis alert patient characteristics	Patients
Age category
< 1 y	1 (3%)
≥ 1–4 y	4 (14%)
≥ 4–12 y	6 (21%)
≥ 12–18 y	13 (45%)
≥ 18 y	5 (17%)
Gender
Female	16 (55%)
Male	13 (45%)
Sepsis criteria[a]
Tachycardia	25 (86%)
Tachypnea	25 (86%)
Temperature abnormality	20 (69%)
Skin abnormality	20 (69%)
Capillary refill abnormality	18 (62%)
High-risk condition	18 (62%)
Concern for infection/temperature abnormality	17 (59%)
Hypotension	15 (52%)
Pulse abnormality	4 (14%)

^a Patients can meet multiple criteria.

Of the 29 patients, the most common sepsis alert criteria included 25 (86%) patients with tachycardia and 25 (86%) patients with tachypnea. The next most common sepsis alert criteria included 20 (69%) patients with a temperature abnormality, 20 (69%) patients with a skin abnormality, 18 (62%) patients with a documented high-risk condition, 18 (62%) patients with a capillary refill abnormality, and 17(59%) patients who had a concern for historical temperature abnormality or infection as documented by the nurse during the intake exam. Hypotension was present in 15 (52%) patients and 4 (14%) of the 29 patients met pulse abnormality criteria.

Discussion

Sepsis continues to be a leading cause of morbidity and mortality worldwide,[16] despite this, the clinical presentation of sepsis is nonspecific and dependent on synthesis of varied clinical information making identification of septic patients challenging. Widely accepted practice has been to identify sepsis through manual application of paper screening tools. In 2015, our institution adopted a modified version of the sepsis trigger/identification tool from the Pediatric Septic Shock Collaborative of the American Academy of Pediatrics. While the work of this early collaborative was vital in the development of a tool that supports both the identification of patients with signs and symptoms of sepsis and offers the opportunity to intervene during the “golden hour” of resuscitation, the utility of the paper trigger tool depends upon routine and repeated application of the tool and may miss opportunities for early sepsis detection. It also allows user bias such as the clinician only applying the tool to patients they already suspect to be septic. Unless reapplied throughout the ED stay, it has the potential to miss patients who develop sepsis during their encounter. The manual tool only includes those patients where there is known fever, hypothermia, or concern for infection on ED presentation.

The increasing and widespread use of EHRs offers a unique way to intervene early in a patient encounter. On trend with other EDs throughout the nation, we wanted to leverage the EHR's ability to recognize discrete documentation of sepsis criteria and prompt notification to ED providers early and reliably, and eliminate user variability and clinician bias. Additionally, we wanted this process to not incur additional burden on provider workflow. Previously published attempts at creating electronic sepsis screening tools were limited due to need for additional documentation, a single assessment that did not update throughout the encounter,[17] [18] or a need for reassessment to look for specific subsequent criteria.[19] [20] Our tool does not introduce or require additional documentation with the exception of a single question during triage. Since the tool is electronic and automated, it assesses all patients on arrival to the ED and on instant of documentation throughout an ED encounter while still allowing providers to continue with their routine clinical workflows.

The electronic sepsis screening tool was found to be as accurate as a paper, manual process. Based on the validation phase of our design, the electronic tool identified 100% (n = 29 over an 8-week testing period) of patients flagged by the manual process which included a wide variety of patients across age groups and medical complexities. In addition to accurate identification, the electronic tool identified patients 68 minutes earlier in the ED encounter when compared with the manual process. The American College of Critical Care Medicine emphasizes early, goal-directed therapy within the first hour of identification,[5] as early detection leads to improved patient outcomes through earlier fluid resuscitation and antibiotic therapy.[6] Our automated, electronic sepsis screening tool supports this goal. This statistically significant finding in the validation phase of our work may be the result of consistent and repeated application of the electronic tool independent of the patient's clinical status. It removes the limitations to identification of sepsis that could previously have been explained by timing of provider reevaluation or communication of changes in patient status and ultimately results in timely notification of a concern for sepsis.

In the informatics literature, Embi et al found that providers have decreasing response rates to EHR-based system alerts over time.[21] Accordingly, we sought to create an alerting system that maintains an appropriate level of concern without an unwarranted number of alerts. For this reason, we created a tool that minimizes excessive alerts while maintaining a high level of sensitivity. By doing this, we maximize the screening capability of the electronic sepsis screening tool.

Limitations

This work was completed at a single institution, and as such, the generalizability of these results may be limited by differences in population. However, the population at our institution typically includes a broad mix of medical and surgical patients in an ED that routinely sees nearly 90,000 patients per year.

Another potential limitation of this process is ensuring the accuracy of the patient's medical history as entered into the medical record. Our institution is unique in that the vast majority of patients with high-risk conditions in the geographic area are seen at our institution, so the historical medical information is typically preexisting in the EHR; however, in other geographic settings where the patient may not be primarily followed at that institution, information may be missing from the record and thus not used as dynamically to calculate a risk for sepsis.

Finally, this tool is intended as a replacement of a manual sepsis screening tool, and patients who do not meet the clinical criteria for sepsis may be missed by both the manual and electronic sepsis screening tools.

Conclusion

The automated, electronic sepsis screening tool is as accurate as a validated manual process and identified patients at risk for sepsis significantly earlier than the manual process. Deployment of this electronic tool has the capability to improve timely sepsis detection and management of patients at risk for sepsis without requiring additional documentation by providers. Importantly, this electronic sepsis screening tool keeps providers in their routine clinical workflows and does not introduce bias.

Multiple Choice Questions

1. Which of the following methods validated the performance of the new electronic screening tool?

   • Retrospective manual chart review.

   • EHR data abstraction using business intelligence software.

   • Deployment of the tool in the live environment to end users.

   • Requiring additional sepsis assessments in the provider workflows.

   Correct Answer: The correct answer is option b. We used Qlikview, a business intelligence software provided by Qlik to evaluate the presence of documented sepsis criteria for all ED encounters at our institution over the 15-month period from January 1, 2016 to March 31, 2017. We focused on ensuring all historical patients documented as a sepsis alert would have been captured by the electronic sepsis screening tool.

2. Which of the following criteria in the EHR was missing as a discrete data element during the crosswalk analysis?

   • Vital sign abnormalities.

   • Abnormalities in physical assessment (i.e., capillary refill, pulse abnormality).

   • Patient information related to infection and/or temperature abnormalities.

   • History of high-risk conditions that place the patient at increased risk for sepsis.

   Correct Answer: The correct answer is option c. After the team completed the crosswalk analysis, all criteria in the paper trigger tool were found as discrete fields in the current EHR with the exception of historical temperature abnormality or provider concern for infection (see [Fig. 2]).

Conflict of Interest

None.

Protection of Human and Animal Subjects

Human and/or animal subjects were not included in this project.

• References

• 1 Hartman ME, Linde-Zwirble WT, Angus DC, Watson RS. Trends in the epidemiology of pediatric severe sepsis. Pediatr Crit Care Med 2013; 14 (07) 686-693
• 2 WHO adopts resolution on sepsis. Jena, Germany: Global Sepsis Alliance, May 26, 2017. Available at: https://www.global-sepsis-alliance.org/news/2017/5/26/wha-adopts-resolution-on-sepsis . Accessed October 20, 2018
• 3 Balamuth F, Weiss SL, Neuman MI. , et al. Pediatric severe sepsis in U.S. children's hospitals. Pediatr Crit Care Med 2014; 15 (09) 798-805
• 4 Watson RS, Carcillo JA, Linde-Zwirble WT, Clermont G, Lidicker J, Angus DC. The epidemiology of severe sepsis in children in the United States. Am J Respir Crit Care Med 2003; 167 (05) 695-701
• 5 Weiss SL, Fitzgerald JC, Balamuth F. , et al. Delayed antimicrobial therapy increases mortality and organ dysfunction duration in pediatric sepsis. Crit Care Med 2014; 42 (11) 2409-2417
• 6 Davis AL, Carcillo JA, Aneja RK. , et al. American College of Critical Care Medicine Clinical Practice Parameters for Hemodynamic Support of Pediatric and Neonatal Septic Shock. Crit Care Med 2017; 45 (06) 1061-1093
• 7 Rivers E, Nguyen B, Havstad S. , et al; Early Goal-Directed Therapy Collaborative Group. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001; 345 (19) 1368-1377
• 8 Paul R, Melendez E, Stack A, Capraro A, Monuteaux M, Neuman MI. Improving adherence to PALS septic shock guidelines. Pediatrics 2014; 133 (05) e1358-e1366
• 9 Lane RD, Funai T, Reeder R, Larsen GY. High reliability pediatric septic shock quality improvement initiative and decreasing mortality. Pediatrics 2016; 138 (04) e20154153
• 10 Akcan Arikan A, Williams EA, Graf JM, Kennedy CE, Patel B, Cruz AT. Resuscitation bundle in pediatric shock decreases acute kidney injury and improves outcomes. J Pediatr 2015; 167 (06) 1301-1305
• 11 Cruz AT, Perry AM, Williams EA, Graf JM, Wuestner ER, Patel B. Implementation of goal-directed therapy for children with suspected sepsis in the emergency department. Pediatrics 2011; 127 (03) e758-e766
• 12 Larsen GY, Mecham N, Greenberg R. An emergency department septic shock protocol and care guideline for children initiated at triage. Pediatrics 2011; 127 (06) e1585-e1592
• 13 Balamuth F, Alpern ER, Abbadessa MK. , et al. Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign–based electronic alert and bedside clinician identification. Ann Emerg Med 2017; 70 (06) 759-768
• 14 The Algorithm Alert Sepsis Collaborative. Available at: http://pedsreadytoolkit.com/wp-content/uploads/2017/03/Triage-Trigger-Tool.pdf . Accessed May 1, 2015
• 15 Balamuth F, Weiss SL, Hall M. , et al. Identifying pediatric severe sepsis and septic shock: accuracy of diagnosis codes. J Pediatr 2015; 167 (06) 1295-300.e4
• 16 Watson RS, Carcillo JA. Scope and epidemiology of pediatric sepsis. Pediatr Crit Care Med 2005; 6 (3, Suppl): S3-S5
• 17 Keep JW, Messmer AS, Sladden R. , et al. National early warning score at emergency department triage may allow earlier identification of patients with severe sepsis and septic shock: a retrospective observational study. Emerg Med J 2016; 33 (01) 37-41
• 18 Hayden GE, Tuuri RE, Scott R. , et al. Triage sepsis alert and sepsis protocol lower times to fluids and antibiotics in the ED. Am J Emerg Med 2016; 34 (01) 1-9
• 19 Cruz AT, Williams EA, Graf JM. , et al. Test characteristics of an automated age- and temperature-adjusted tachycardia alert in pediatric septic shock. Pediatr Emerg Care 2012; 28 (09) 889-894
• 20 Sepanski RJ, Godambe SA, Mangum CD, Bovat CS, Zaritsky AL, Shah SH. Designing a pediatric severe sepsis screening tool. Front Pediatr 2014; 2 (56) 56
• 21 Embi PJ, Leonard AC. Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Inform Assoc 2012; 19 (e1): e145-e148