Keywords
clinical decision support systems - alert fatigue - patient safety - clinical governance
Background and Significance
Background and Significance
Intelligently designed clinical decision support (CDS) has long been held as one of
the most promising benefits of the electronic health record (EHR). Indeed, implementation
of CDS was one of the core objectives of the Centers for Medicare and Medicaid Services
Meaningful Use EHR incentive programs. To be effective, though, CDS needs to be carefully
and thoughtfully designed. Recognizing that effective CDS must influence clinician
behavior and actions to achieve the desired outcome, key tenets to CDS design have
been identified that principally relate to attention to clinician workflows and user-centered
principles in order to adhere to the “five rights” of CDS.[1]
[2] Failure to follow these best practices for CDS can not only lead to a failure to
meet the intended objective but can also exacerbate “alert fatigue,” a phenomenon
where increasing exposures to alerts desensitizes clinicians to them.[3]
[4]
[5]
[6] This may not only contribute to clinician frustration and burnout but also is inherently
a safety risk as it may lead clinicians to ignore other critical alerts.[7]
[8] Additionally, interruptions in clinical workflow caused by alerts can, in isolation,
be a safety risk with one study showing that interruptions increased the chance of
procedural failures and clinical errors in medication administrations by nurses by
greater than 12%.[9]
Given the inherent risks of alert fatigue, there is a need for careful attention to
best practices to ensure effective implementation of CDS that minimizes the risk of
unnecessary interruptive alerts. Prior literature supports the benefits of applying
human factors' principles and incorporating usability testing during the design and
implementation of CDS systems.[10]
[11]
[12] Additionally, evaluating the frequency with which criteria for alert firing are
met on retrospective data or through prospective evaluation “in the background” can
allow for refinement of alert triggers prior to introduction into clinical care.[13]
[14]
It is important though to not only focus on best practices during the initial design
of a CDS intervention but also following implementation, particularly for alerts that
interrupt a clinician in their workflow. Previous literature on de-implementation
of interruptive alerts has described the efficacy of interventions to reduce alert
firings, with a particular focus on the key role of governance as well as data monitoring
to evaluate effectiveness.[15]
[16]
[17]
[18]
[19] Application of quality improvement (QI) methodologies to the problem of CDS alert
fatigue has also proven beneficial, but these studies have focused primarily on improving
metrics on alert data.[15]
[19] One of the ideal best practices though is to define and track the clinical outcome
or goal that an alert is intended to achieve, but often these distal measures are
more challenging and resource-intensive to obtain.[17] Understanding these measures, however, can allow for targeted adjustments of the
CDS and to comparatively evaluate noninterruptive alternatives to see if they can
accomplish the same clinical goals. Here, we will describe how defining alert and
clinical metrics allowed us to introduce iterative data-driven improvements that reduced
the burden of a single interruptive alert while achieving better clinical endpoints.
Objectives
In this report, aligned with Standards for QUality Improvement Reporting Excellence
(SQUIRE) 2.0 reporting guidelines ([Supplementary Table 1]), [20] we describe the de-implementation of a single frequently firing interruptive Best
Practice Advisory (BPA) at our institution. We undertook this effort as part of a
broader ongoing QI project to overhaul our CDS governance and reduce the interruptive
alert burden faced by our clinicians. We are focusing our QI work on evaluation of
“open chart” BPAs (those that are triggered immediately upon opening a patient's chart)
as we have identified these as particularly problematic based on review of our alert
data and recognition that they often appear at the “wrong time” in a clinician's workflow.
The “open chart” BPA we focus on here was implemented at the start of the coronavirus
disease 2019 (COVID) pandemic and intended to alert clinicians to order appropriate
infection prevention (IP) precautions when a patient with symptoms possibly consistent
with COVID (based on screening questions on presentation [[Supplementary Fig. S1], available in the online version only]) arrived at an emergency department (ED),
inpatient, or ambulatory setting ([Fig. 1]). At the time of implementation, the goal of this alert, which was designed in a
red color scheme to represent it as “critical” to clinicians, was to promptly identify
any patient with one or more symptoms of possible COVID in order to alert staff, support
allocation of personal protective equipment (PPE), and allow for efficient patient
cohorting to limit nosocomial spread. Due to the public health emergency, the implementation
was critical and quick, but because of this, there was no plan established at the
time of implementation related to evaluation, optimization, or future de-implementation.
Thus, even as the COVID pandemic evolved with PPE shortages resolving, vaccinations
becoming available to protect staff, and specialized COVID units in our hospital closing,
this highly sensitive and nonspecific BPA remained. This mismatch with the evolving
clinical realities of COVID coupled with the uniquely burdensome nature of this alert
(it was the most frequently firing interruptive BPA across our health system at the
outset of our project) prompted a need for focused attention to address this problematic
BPA.
Fig. 1 Original interruptive BPA that appeared on opening a patient's chart within our EHR
for patients symptomatic for COVID. BPA, Best Practice Advisory; COVID, coronavirus
disease 2019; EHR, electronic health record.
This report aims to describe our improvement efforts to reduce the alert burden of
this particular BPA and our evaluation of the success of our approach. Informatics
solutions included data monitoring and the development of an alternative passive CDS
system. There was also a need for significant interdisciplinary collaboration and
communication across our health system. When we on the clinical informatics team initially
proposed retiring this BPA, operational team members from IP raised concerns about
the potential negative impact on rates of appropriate infection precautions ordering,
particularly during inpatient admissions where cohorting of potential COVID patients
remained a priority. Collaboratively defining these updated clinical requirements
with operations was a key step and led us to develop two-fold aims for our project
and report. First, from an informatics perspective, we wanted to monitor the interruptive
alert burden and specifically reduce the total number of interruptive alerts as well
as the rate of interruptive alerts per encounter based on prior reports on the utility
of these metrics in CDS evaluation.[21] From an operational perspective, our key goal was to monitor, and hopefully improve,
the percentage of admitted patients with provider-identified symptoms concerning for
COVID who had appropriate droplet, contact, and airborne precautions (COVID precautions)
ordered as well as the average time between when a severe acute respiratory syndrome
coronavirus 2(SARS-CoV-2) polymerase chain reaction (PCR) test was ordered in a patient
with symptoms and when they had COVID precautions ordered.
Methods
Context
University of Rochester Medical Center is an academic health system comprising six
hospitals and one pediatric hospital in central upstate New York, including Strong
Memorial Hospital, our flagship quaternary care center with 886 beds. All of our hospitals
share the same instance of the Epic EHR (Epic Systems, Verona, WI). Our clinical informatics
and CDS governance structures have evolved significantly over the last several years
with broader representation and support for provider and nursing clinical informaticists.
Interventions
Our CDS workgroup, which includes representatives from provider and nursing informatics
as well as Information Systems Division analysts, led this effort with input from
operational stakeholders from Clinical Leadership, Quality and Safety, Infectious
Disease (ID), and IP. Broad operational engagement and support from across all our
hospitals was required which was accomplished via emails and attendance at virtual
meetings with IP leadership from all hospitals to explain the problem of “alert fatigue,”
review the baseline data related specifically to the COVID BPA, and to discuss proposed
adjustments to the CDS build. Based on interdisciplinary input, we undertook iterative
build changes, in alignment with the Model for Improvement methodology and grounded
in the CDS “five rights” principles ([Fig. 2]).[22] To address clinician and staff concerns related to inappropriate appearance of the
alert to individuals not primarily participating in the patient's care, our first
intervention was to add a “Not on Care Team” acknowledgment option in late October
2022 across all care settings. Decreasing the sensitivity of the alert to reduce the
frequency of firing was the focus of our next adjustment in early February 2023. To
accomplish this, we increased the minimum number of symptoms from the triage screening
that were required to trigger the BPA from a single symptom to two symptoms. The final
CDS build adjustment (to-date) in late March 2023 was to develop passive CDS and fully
de-implement the interruptive BPA in the ED, inpatient, and urgent care settings.
Due to operational need in the ambulatory care areas to quickly identify patients
with possible COVID for rapid rooming during check-in, the BPA remained interruptive
in our outpatient clinics. With input from IP, ID, and operational provider leadership,
we developed passive CDS for nonambulatory areas in the form of a new rule-based order
panel that evaluated for the presence of COVID precautions orders when a provider
was ordering a SARS-CoV-2 PCR and indicated that the patient was symptomatic ([Fig. 3]). If the patient was deemed symptomatic and did not yet have COVID precautions ordered,
the appropriate precautions orders were automatically added to the SARS-CoV-2 PCR
order. If they already had an active precautions order, it did not appear in the order
panel. The goal was to ensure that by the time the provider identified a patient as
sufficiently symptomatic to require a PCR, appropriate precautions were ordered and
to particularly ensure this occurred among patients once admitted to the hospital
to allow for appropriate and efficient cohorting and placement decisions.
Fig. 2 Schematic of iterative CDS design changes made through the course of our initiative
and their alignment with the “five rights” framework of CDS. BPA, Best Practice Advisory;
CDS, clinical decision support; COVID, coronavirus disease 2019; PCR, polymerase chain
reaction.
Fig. 3 Rule-based order panel that linked SARS-CoV-2 PCR orders on symptomatic patients
with COVID precautions orders when they had not yet been placed. Of note, prior to
seeing the view that is shown, the ordering provider indicated via single-select whether
the patient was symptomatic or asymptomatic. COVID, coronavirus disease 2019; IP,
infection prevention; PCR, polymerase chain reaction.
Data Collection and Analysis
EHR data were extracted via Structured Query Language queries of our Clarity database.
We extracted and analyzed aggregated de-identified data in Microsoft Excel using QI
Macros (www.qimacros.com, Denver, CO) to generate Statistical Process Control (SPC) charts. We chose specific
SPC charts for each metric based on review of QI analytics standards and used standard
rules to identify special cause variation, particularly that a centerline shift occurred
when eight consecutive points were above or below the mean or median.[22]
[23] Our primary informatics outcome metrics were the number of interruptive alerts per
week and the average interruptive alerts per encounter per week across all care settings.
We tracked these metrics on run charts for all BPAs as well as for the COVID BPA specifically
but we only analyzed the COVID BPA using special cause rules given the potential for
external factors to impact the overall alert data. Our clinical metrics included the
percent of admitted patients (inpatient or observation status) with symptoms of COVID
(per provider assessment in the PCR order) who had appropriate infection precautions
ordered. We excluded precautions orders that were placed more than 48 hours before
or after the PCR order based on an assumption that these were due to separate episodes
of concern for possible COVID during the hospitalization. We tracked this metric over
time on a percent nonconforming attribute SPC chart (p chart). Additionally, to assess timely ordering of COVID precautions, we monitored
the average time between when a COVID PCR was ordered in a patient with symptoms and
when they had appropriate infection precautions ordered on an individual-moving range
SPC chart (X-mR chart).
Results
Informatics Metrics
Our total interruptive alert volume and interruptive alert per encounter rate tracked
by week over our study period are shown in [Figs. 4] and [5], respectively. At baseline, there was a median of 8,206 COVID BPA alerts per week
([Fig. 4]) and the median COVID BPA alert per encounter rate was 0.36 ([Fig. 5]). With our initial intervention of adding an additional acknowledgment option, there
was an increase in our median for both metrics. Of note, the alert volume appeared
to be increasing even prior to our intervention, which we suspect was related to a
concurrent increase in encounter volume (data not shown) as well as contemporaneous
implementation of electronic check-in via in-office kiosks in some of our ambulatory
areas so that patients were now directly answering screening questions rather than
being verbally asked by staff who then entered the data. It is notable though that
there was a more sustained and notable increase when we implemented our first intervention.
We believe this was due to the fact that the “Not on Care Team” button was added not
to suppress the BPA, but to allow users who were seeing it inappropriately to bypass
it, so subsequent actions to open the chart caused the BPA to refire. Once we raised
the threshold for the interruptive BPA to fire from one symptom to two, we observed
a sustained decrease in our median for both metrics. Finally, when we de-implemented
the interruptive COVID BPA in the acute care settings and replaced it with passive
CDS in those areas, we observed a final downward shift in our median for both metrics
to 1,449 alerts per week and 0.07 alerts per encounter. This amounted to an over 80%
reduction in the alert burden for both metrics. In looking at the total alert data
and alert per encounter data, it appears that the trends mirrored the COVID BPA data
over time, which likely reflects the fact that this BPA was one of our highest firing
interruptive alerts.
Fig. 4 Run chart displaying Total Alerts per week for all Alerts, in black, and COVID Alerts,
in blue. The median Total Alert volume for the COVID Alerts is shown in teal. Boxed
annotations represent interventions implemented at the designated time points. BPA,
Best Practice Advisory; COVID, coronavirus disease 2019; ED, emergency department.
Fig. 5 Run chart displaying Alert per Encounter rate for all Alerts, in black, and COVID
Alerts, in blue. The median Alert per Encounter rate for the COVID Alerts is shown
in teal. Boxed annotations represent interventions implemented at the designated time
points. BPA, Best Practice Advisory; COVID, coronavirus disease 2019; ED, emergency
department.
Clinical Metrics
The primary IP goal in our hospital was to ensure that hospitalized patients symptomatic
for COVID were under appropriate infection precautions to limit the spread of COVID
to staff and unaffected patients. At the outset of our monitoring period with the
original BPA design, the mean percentage of symptomatic admitted patients with COVID
precautions orders was 23% ([Fig. 6]). The initial interventions that were implemented were primarily focused on reducing
the alert burden for clinicians and it was notable that with our second intervention,
increasing the specificity of the BPA trigger to two symptoms rather than just one,
the mean percentage of symptomatic patients with COVID precautions ordered decreased
to 14%. Ultimately, however, when we transitioned the CDS from the interruptive BPA
to a noninterruptive order panel, we observed a significant upward shift in our mean
to 61%, an absolute increase of 47% and 38% higher than baseline.
Fig. 6 P-chart showing the percentage of hospitalized patients identified as symptomatic
for COVID-19 who had COVID precautions ordered within 48 hours of PCR order in blue.
The mean rate is shown in teal with upper and lower control limits (±3 SD from centerline)
in red. Boxed annotations represent interventions implemented at the designated time
points. COVID, coronavirus disease 2019; PCR, polymerase chain reaction; SD, standard
deviation; UCL, upper control limit; LCL, lower control limit.
Our second clinical metric was the average time between when a SARS-CoV-2 PCR for
a symptomatic patient was ordered and when appropriate COVID precautions were ordered.
At baseline, the average time between the PCR and precautions order was 19.7 minutes
but there was significant variability week-to-week as evidenced by wide control limits
on our X chart and a baseline moving range of 65 minutes in our moving range (mR)
chart in [Fig. 7]. There was no special cause variation observed on our X chart directly related to
our initial interventions though there was a period of special cause with 11 consecutive
points above the mean that began in December 2022 midway between our first intervention
and our second. As this did not correlate specifically with one of our interventions,
we did not shift our centerline and suspect this may have been due to increased patient
volume or acuity at this time. Following replacement of the interruptive BPA with
the rule-based order panel in March of 2023, we observed an immediate response in
our data, with a downward shift in our mean to −1.3 minutes indicating that the precautions
orders were being placed, on average, slightly ahead of the PCR orders. Additionally,
there was a substantial reduction in the variability week-to-week as apparent by the
narrowing of our control limits and the reduction in our moving range to 18.2 minutes.
Fig. 7 X-mR chart showing the time between SARS-CoV-2 PCR orders and COVID precautions orders
among symptomatic patients. The Individuals Chart (X) is on top shows the average
difference in minutes among all orders in a given week in blue, the mean of these
averages in teal, and the upper and lower control limits (±3 SD from centerline) in
red. The moving range (mR) chart shows the absolute difference between successive
data points over time in blue, with the average range of variability in teal, and
the upper and lower control limits (±3 SD from centerline) in red. Boxed annotations
represent interventions implemented at the designated time points. COVID, coronavirus
disease 2019; SD, standard deviation; UCL, upper control limit; LCL, lower control
limit.
Discussion
Summary and Interpretation
We aimed in this initiative to reduce the alert burden of a single problematic BPA
while supporting the clinical need for appropriate and prompt ordering of infection
precautions in admitted patients with potential COVID. Through iterative design changes
to better align with the “five rights” of CDS, we were able to reduce the overall
alert volume and alert per encounter rate by over 80% in our health system. Furthermore,
by replacing an interruptive BPA with a rule-based order panel in acute care settings,
we also significantly improved the rate of appropriate infection precautions being
ordered for admitted patients with potential COVID and reduced the time between a
SARS-CoV-2 PCR and COVID precautions orders in these patients, which were key clinical
priorities for our infection preventionists and hospital administration.
CDS has become synonymous with interruptive BPA alerts at many institutions. Often,
this is due to a lack of awareness by operational stakeholders requesting CDS of the
full scope of decision support options within the EHR or the assumption that a “pop
up” is the most effective means available. In general, clinicians prefer “nudges”
rather than interruptive alerts,[24] and passive decision support, while perhaps not entirely benign,[25]
[26] does not pose the same alert fatigue risk as an interruptive alert that forcibly
disrupts a clinician's workflow. While there may be high-risk circumstances necessitating
an interruptive alert, it is beneficial to educate operational partners requesting
a BPA about available CDS options and to take a more user-centered approach by aiming
to use noninterruptive methods whenever feasible. Additionally, if interruptive CDS
mechanisms are chosen, then it is critical to follow metrics on alert burden and clinical
outcomes and adjust the CDS design in a data-driven fashion. Doing so provides a foundation
for application of QI methodologies to not only reduce alert burden and clinician
disruptions, as others have reported,[15]
[19] but also, as we described here, to achieve improved clinical endpoints via iteratively
designing CDS towards the “five rights.”
Limitations
There are limitations to our work that should be acknowledged, however. First, the
success of our approach was driven by institutional investment in clinical informaticists
who are skilled in change management, clinical workflow analysis, and technical feasibility
of CDS. This investment was recent at our institution, and we recognize that it is
not universal which may limit the generalizability of this work. Additionally, our
passive CDS intervention only targeted admitted patients whose SARS-CoV-2 PCR was
ordered within that encounter. As some patients might have been tested prior to ED
or hospital arrival, these patients would have been missed by our intervention and
by our data which use provider identification of patients as symptomatic in the PCR
order to identify those with symptoms. Finally, we are keenly aware that we could
not control for other systems or technical forces that might have influenced the trends
in our data. For example, provider ordering behavior surrounding the timing of SARS-CoV-2
PCR orders could have been altered by evolving vaccination rates and the prevalence
of COVID in our community. It is notable as well that the overall alert trend did
appear to rise and fall over the course of our data collection and while that was
largely related to the COVID BPA trends, which appeared to be driven by our interventions,
it could suggest that other factors may have been at play.
Additionally, while improvements in both our informatics and clinical metrics were
clear and significant, there is still room for further improvement. The interruptive
BPA remains active in the ambulatory setting and one key next step is to work on devising
alternative CDS that would be less interruptive while still addressing the need to
identify potential COVID patients for rapid rooming. Additionally, with 39% of symptomatic
admitted patients still not having precautions orders, we clearly do have room for
further optimization of our passive CDS in the acute care setting. One beneficial
next step would be to carry out structured usability assessments to identify possible
sources of error or poor usability that may be a factor, as this has previously been
shown to be beneficial in improving decision support systems.[10]
[11] Further data analysis of cases where precautions were not appropriately ordered
would also be beneficial to better identify targets for optimizations or refinement
to our CDS, or perhaps additional appropriate clinical exceptions that need to be
incorporated.
Conclusion
In conclusion, we were able to successfully reduce the alert burden while simultaneously
improving clinical outcomes by transitioning from an interruptive BPA to a noninterruptive
rule-based order panel. Before closing, though, it is important to recognize the time
and effort that was required to de-implement this single BPA in acute care settings
at our institution. As with other areas of medicine where de-implementation of unnecessary
or nonevidence-based practices can be particularly challenging,[27] the work to de-implement CDS is not insignificant. Our workgroup had identified
this BPA as problematic from the standpoint of clinician burden well over a year before
we were able to de-implement it in our inpatient, ED, and urgent care areas. Beyond
the work to design and build the interventions described in this report, there were
countless emails and numerous hours of meetings with operational stakeholders to achieve
buy-in across our multihospital health system. This highlights the challenge of dismantling
entrenched, established practices in general, but particularly when it comes to CDS
within the EHR that is aimed at promoting safe care or reducing the risk of patient
harm. The default assumption, even when the impact is not measured, is that a BPA,
once embedded within clinical workflows, must be doing something and negative outcomes will result if it is taken away. This presumption and the time
and effort required to de-implement this BPA further emphasizes the crucial importance
of thoughtful user-centered design and upfront CDS governance at the time of implementation
of an interruptive BPA. Only with careful governance, including a plan for prospective
evaluation and planned de-implementation of ineffective alerts, will we arrive at
intelligent and effective CDS that achieves intended clinical outcomes without overburdening
clinicians.
Clinical Relevance Statement
Clinical Relevance Statement
Given the risks of alert fatigue, governance, evaluation, and optimization are critical
when implementing CDS in the EHR. Here, we report on how we replaced an interruptive
alert with noninterruptive CDS that achieved better clinical endpoints with reduced
alert burden.
Multiple Choice Questions
Multiple Choice Questions
-
At what stage of clinical decision support design and implementation is it important
to consider a governance plan, including how to evaluate CDS efficacy, alert burden,
and outcomes?
Correct answer: b. Prior to implementation. Developing a governance plan prior to
implementation of a new CDS intervention, particularly when an interruptive alert,
is best practice. It is helpful at the outset of a request for a new CDS tool to clearly
define the clinical outcome that it is aiming to achieve as well as a plan to measure
that outcome prospectively following implementation, along with metrics related to
the alert burden.
-
Your institution's chief quality officer (CQO) requests a new “pop-up” to appear to
ordering providers every time an expiring medication is about to fall off the patient's
medication list after a patient experienced a seizure due to an antiepileptic medication
auto-discontinuing. She indicates this will help to prevent this from occurring in
the future. As the informaticist assigned to assist with this request, what are key
considerations to discuss with the CQO?
-
Risks of alert fatigue due to a new interruptive alert
-
Alternative decision support systems that are nonpassive besides a “pop-up”
-
Plan for prospective evaluation of clinical outcome and alert burden metrics
-
All of the above.
Correct answer: d. All of the above. It is important to partner with operational stakeholders
who request new clinical decision support systems to educate them about the risks
of alert fatigue and the alternatives to interruptive alerts. Collaboratively developing
a plan for prospective evaluation of the efficacy following implementation is also
critical to facilitate data-driven adjustments to the CDS to ensure clinical efficacy
with minimal alert burden.
