Addressing Alert Fatigue by Replacing a Burdensome Interruptive Alert with Passive Clinical Decision Support

Abstract

Background Recognizing that alert fatigue poses risks to patient safety and clinician wellness, there is a growing emphasis on evaluation and governance of electronic health record clinical decision support (CDS). This is particularly critical for interruptive alerts to ensure that they achieve desired clinical outcomes while minimizing the burden on clinicians. This study describes an improvement effort to address a problematic interruptive alert intended to notify clinicians about patients needing coronavirus disease 2019 (COVID) precautions and how we collaborated with operational leaders to develop an alternative passive CDS system in acute care areas.

Objectives Our dual aim was to reduce the alert burden by redesigning the CDS to adhere to best practices for decision support while also improving the percent of admitted patients with symptoms of possible COVID who had appropriate and timely infection precautions orders.

Methods Iterative changes to CDS design included adjustment to alert triggers and acknowledgment reasons and development of a noninterruptive rule-based order panel for acute care areas. Data on alert burden and appropriate precautions orders on symptomatic admitted patients were followed over time on run and attribute (p) and individuals-moving range control charts.

Results At baseline, the COVID alert fired on average 8,206 times per week with an alert per encounter rate of 0.36. After our interventions, the alerts per week decreased to 1,449 and alerts per encounter to 0.07 equating to an 80% reduction for both metrics. Concurrently, the percentage of symptomatic admitted patients with COVID precautions ordered increased from 23 to 61% with a reduction in the mean time between COVID test and precautions orders from 19.7 to −1.3 minutes.

Conclusion CDS governance, partnering with operational stakeholders, and iterative design led to successful replacement of a frequently firing interruptive alert with less burdensome passive CDS that improved timely ordering of COVID precautions.

 

Scroll to top

Keywords

clinical decision support systems - alert fatigue - patient safety - clinical governance

Background and Significance

Intelligently designed clinical decision support (CDS) has long been held as one of the most promising benefits of the electronic health record (EHR). Indeed, implementation of CDS was one of the core objectives of the Centers for Medicare and Medicaid Services Meaningful Use EHR incentive programs. To be effective, though, CDS needs to be carefully and thoughtfully designed. Recognizing that effective CDS must influence clinician behavior and actions to achieve the desired outcome, key tenets to CDS design have been identified that principally relate to attention to clinician workflows and user-centered principles in order to adhere to the “five rights” of CDS.[1] [2] Failure to follow these best practices for CDS can not only lead to a failure to meet the intended objective but can also exacerbate “alert fatigue,” a phenomenon where increasing exposures to alerts desensitizes clinicians to them.[3] [4] [5] [6] This may not only contribute to clinician frustration and burnout but also is inherently a safety risk as it may lead clinicians to ignore other critical alerts.[7] [8] Additionally, interruptions in clinical workflow caused by alerts can, in isolation, be a safety risk with one study showing that interruptions increased the chance of procedural failures and clinical errors in medication administrations by nurses by greater than 12%.[9]

Given the inherent risks of alert fatigue, there is a need for careful attention to best practices to ensure effective implementation of CDS that minimizes the risk of unnecessary interruptive alerts. Prior literature supports the benefits of applying human factors' principles and incorporating usability testing during the design and implementation of CDS systems.[10] [11] [12] Additionally, evaluating the frequency with which criteria for alert firing are met on retrospective data or through prospective evaluation “in the background” can allow for refinement of alert triggers prior to introduction into clinical care.[13] [14]

It is important though to not only focus on best practices during the initial design of a CDS intervention but also following implementation, particularly for alerts that interrupt a clinician in their workflow. Previous literature on de-implementation of interruptive alerts has described the efficacy of interventions to reduce alert firings, with a particular focus on the key role of governance as well as data monitoring to evaluate effectiveness.[15] [16] [17] [18] [19] Application of quality improvement (QI) methodologies to the problem of CDS alert fatigue has also proven beneficial, but these studies have focused primarily on improving metrics on alert data.[15] [19] One of the ideal best practices though is to define and track the clinical outcome or goal that an alert is intended to achieve, but often these distal measures are more challenging and resource-intensive to obtain.[17] Understanding these measures, however, can allow for targeted adjustments of the CDS and to comparatively evaluate noninterruptive alternatives to see if they can accomplish the same clinical goals. Here, we will describe how defining alert and clinical metrics allowed us to introduce iterative data-driven improvements that reduced the burden of a single interruptive alert while achieving better clinical endpoints.

 

Scroll to top

Objectives

In this report, aligned with Standards for QUality Improvement Reporting Excellence (SQUIRE) 2.0 reporting guidelines ([Supplementary Table 1]), [20] we describe the de-implementation of a single frequently firing interruptive Best Practice Advisory (BPA) at our institution. We undertook this effort as part of a broader ongoing QI project to overhaul our CDS governance and reduce the interruptive alert burden faced by our clinicians. We are focusing our QI work on evaluation of “open chart” BPAs (those that are triggered immediately upon opening a patient's chart) as we have identified these as particularly problematic based on review of our alert data and recognition that they often appear at the “wrong time” in a clinician's workflow. The “open chart” BPA we focus on here was implemented at the start of the coronavirus disease 2019 (COVID) pandemic and intended to alert clinicians to order appropriate infection prevention (IP) precautions when a patient with symptoms possibly consistent with COVID (based on screening questions on presentation [[Supplementary Fig. S1], available in the online version only]) arrived at an emergency department (ED), inpatient, or ambulatory setting ([Fig. 1]). At the time of implementation, the goal of this alert, which was designed in a red color scheme to represent it as “critical” to clinicians, was to promptly identify any patient with one or more symptoms of possible COVID in order to alert staff, support allocation of personal protective equipment (PPE), and allow for efficient patient cohorting to limit nosocomial spread. Due to the public health emergency, the implementation was critical and quick, but because of this, there was no plan established at the time of implementation related to evaluation, optimization, or future de-implementation. Thus, even as the COVID pandemic evolved with PPE shortages resolving, vaccinations becoming available to protect staff, and specialized COVID units in our hospital closing, this highly sensitive and nonspecific BPA remained. This mismatch with the evolving clinical realities of COVID coupled with the uniquely burdensome nature of this alert (it was the most frequently firing interruptive BPA across our health system at the outset of our project) prompted a need for focused attention to address this problematic BPA.

Fig. 1 Original interruptive BPA that appeared on opening a patient's chart within our EHR for patients symptomatic for COVID. BPA, Best Practice Advisory; COVID, coronavirus disease 2019; EHR, electronic health record.

This report aims to describe our improvement efforts to reduce the alert burden of this particular BPA and our evaluation of the success of our approach. Informatics solutions included data monitoring and the development of an alternative passive CDS system. There was also a need for significant interdisciplinary collaboration and communication across our health system. When we on the clinical informatics team initially proposed retiring this BPA, operational team members from IP raised concerns about the potential negative impact on rates of appropriate infection precautions ordering, particularly during inpatient admissions where cohorting of potential COVID patients remained a priority. Collaboratively defining these updated clinical requirements with operations was a key step and led us to develop two-fold aims for our project and report. First, from an informatics perspective, we wanted to monitor the interruptive alert burden and specifically reduce the total number of interruptive alerts as well as the rate of interruptive alerts per encounter based on prior reports on the utility of these metrics in CDS evaluation.[21] From an operational perspective, our key goal was to monitor, and hopefully improve, the percentage of admitted patients with provider-identified symptoms concerning for COVID who had appropriate droplet, contact, and airborne precautions (COVID precautions) ordered as well as the average time between when a severe acute respiratory syndrome coronavirus 2(SARS-CoV-2) polymerase chain reaction (PCR) test was ordered in a patient with symptoms and when they had COVID precautions ordered.

 

Scroll to top

Methods

Context

University of Rochester Medical Center is an academic health system comprising six hospitals and one pediatric hospital in central upstate New York, including Strong Memorial Hospital, our flagship quaternary care center with 886 beds. All of our hospitals share the same instance of the Epic EHR (Epic Systems, Verona, WI). Our clinical informatics and CDS governance structures have evolved significantly over the last several years with broader representation and support for provider and nursing clinical informaticists.

 

Scroll to top

Interventions

Our CDS workgroup, which includes representatives from provider and nursing informatics as well as Information Systems Division analysts, led this effort with input from operational stakeholders from Clinical Leadership, Quality and Safety, Infectious Disease (ID), and IP. Broad operational engagement and support from across all our hospitals was required which was accomplished via emails and attendance at virtual meetings with IP leadership from all hospitals to explain the problem of “alert fatigue,” review the baseline data related specifically to the COVID BPA, and to discuss proposed adjustments to the CDS build. Based on interdisciplinary input, we undertook iterative build changes, in alignment with the Model for Improvement methodology and grounded in the CDS “five rights” principles ([Fig. 2]).[22] To address clinician and staff concerns related to inappropriate appearance of the alert to individuals not primarily participating in the patient's care, our first intervention was to add a “Not on Care Team” acknowledgment option in late October 2022 across all care settings. Decreasing the sensitivity of the alert to reduce the frequency of firing was the focus of our next adjustment in early February 2023. To accomplish this, we increased the minimum number of symptoms from the triage screening that were required to trigger the BPA from a single symptom to two symptoms. The final CDS build adjustment (to-date) in late March 2023 was to develop passive CDS and fully de-implement the interruptive BPA in the ED, inpatient, and urgent care settings. Due to operational need in the ambulatory care areas to quickly identify patients with possible COVID for rapid rooming during check-in, the BPA remained interruptive in our outpatient clinics. With input from IP, ID, and operational provider leadership, we developed passive CDS for nonambulatory areas in the form of a new rule-based order panel that evaluated for the presence of COVID precautions orders when a provider was ordering a SARS-CoV-2 PCR and indicated that the patient was symptomatic ([Fig. 3]). If the patient was deemed symptomatic and did not yet have COVID precautions ordered, the appropriate precautions orders were automatically added to the SARS-CoV-2 PCR order. If they already had an active precautions order, it did not appear in the order panel. The goal was to ensure that by the time the provider identified a patient as sufficiently symptomatic to require a PCR, appropriate precautions were ordered and to particularly ensure this occurred among patients once admitted to the hospital to allow for appropriate and efficient cohorting and placement decisions.

Fig. 2 Schematic of iterative CDS design changes made through the course of our initiative and their alignment with the “five rights” framework of CDS. BPA, Best Practice Advisory; CDS, clinical decision support; COVID, coronavirus disease 2019; PCR, polymerase chain reaction.

Fig. 3 Rule-based order panel that linked SARS-CoV-2 PCR orders on symptomatic patients with COVID precautions orders when they had not yet been placed. Of note, prior to seeing the view that is shown, the ordering provider indicated via single-select whether the patient was symptomatic or asymptomatic. COVID, coronavirus disease 2019; IP, infection prevention; PCR, polymerase chain reaction.

 

Scroll to top

Data Collection and Analysis

EHR data were extracted via Structured Query Language queries of our Clarity database. We extracted and analyzed aggregated de-identified data in Microsoft Excel using QI Macros (www.qimacros.com, Denver, CO) to generate Statistical Process Control (SPC) charts. We chose specific SPC charts for each metric based on review of QI analytics standards and used standard rules to identify special cause variation, particularly that a centerline shift occurred when eight consecutive points were above or below the mean or median.[22] [23] Our primary informatics outcome metrics were the number of interruptive alerts per week and the average interruptive alerts per encounter per week across all care settings. We tracked these metrics on run charts for all BPAs as well as for the COVID BPA specifically but we only analyzed the COVID BPA using special cause rules given the potential for external factors to impact the overall alert data. Our clinical metrics included the percent of admitted patients (inpatient or observation status) with symptoms of COVID (per provider assessment in the PCR order) who had appropriate infection precautions ordered. We excluded precautions orders that were placed more than 48 hours before or after the PCR order based on an assumption that these were due to separate episodes of concern for possible COVID during the hospitalization. We tracked this metric over time on a percent nonconforming attribute SPC chart (p chart). Additionally, to assess timely ordering of COVID precautions, we monitored the average time between when a COVID PCR was ordered in a patient with symptoms and when they had appropriate infection precautions ordered on an individual-moving range SPC chart (X-mR chart).

 

Scroll to top

 

Scroll to top

Results

Informatics Metrics

Our total interruptive alert volume and interruptive alert per encounter rate tracked by week over our study period are shown in [Figs. 4] and [5], respectively. At baseline, there was a median of 8,206 COVID BPA alerts per week ([Fig. 4]) and the median COVID BPA alert per encounter rate was 0.36 ([Fig. 5]). With our initial intervention of adding an additional acknowledgment option, there was an increase in our median for both metrics. Of note, the alert volume appeared to be increasing even prior to our intervention, which we suspect was related to a concurrent increase in encounter volume (data not shown) as well as contemporaneous implementation of electronic check-in via in-office kiosks in some of our ambulatory areas so that patients were now directly answering screening questions rather than being verbally asked by staff who then entered the data. It is notable though that there was a more sustained and notable increase when we implemented our first intervention. We believe this was due to the fact that the “Not on Care Team” button was added not to suppress the BPA, but to allow users who were seeing it inappropriately to bypass it, so subsequent actions to open the chart caused the BPA to refire. Once we raised the threshold for the interruptive BPA to fire from one symptom to two, we observed a sustained decrease in our median for both metrics. Finally, when we de-implemented the interruptive COVID BPA in the acute care settings and replaced it with passive CDS in those areas, we observed a final downward shift in our median for both metrics to 1,449 alerts per week and 0.07 alerts per encounter. This amounted to an over 80% reduction in the alert burden for both metrics. In looking at the total alert data and alert per encounter data, it appears that the trends mirrored the COVID BPA data over time, which likely reflects the fact that this BPA was one of our highest firing interruptive alerts.

Fig. 4 Run chart displaying Total Alerts per week for all Alerts, in black, and COVID Alerts, in blue. The median Total Alert volume for the COVID Alerts is shown in teal. Boxed annotations represent interventions implemented at the designated time points. BPA, Best Practice Advisory; COVID, coronavirus disease 2019; ED, emergency department.

Fig. 5 Run chart displaying Alert per Encounter rate for all Alerts, in black, and COVID Alerts, in blue. The median Alert per Encounter rate for the COVID Alerts is shown in teal. Boxed annotations represent interventions implemented at the designated time points. BPA, Best Practice Advisory; COVID, coronavirus disease 2019; ED, emergency department.

 

Scroll to top

Clinical Metrics

The primary IP goal in our hospital was to ensure that hospitalized patients symptomatic for COVID were under appropriate infection precautions to limit the spread of COVID to staff and unaffected patients. At the outset of our monitoring period with the original BPA design, the mean percentage of symptomatic admitted patients with COVID precautions orders was 23% ([Fig. 6]). The initial interventions that were implemented were primarily focused on reducing the alert burden for clinicians and it was notable that with our second intervention, increasing the specificity of the BPA trigger to two symptoms rather than just one, the mean percentage of symptomatic patients with COVID precautions ordered decreased to 14%. Ultimately, however, when we transitioned the CDS from the interruptive BPA to a noninterruptive order panel, we observed a significant upward shift in our mean to 61%, an absolute increase of 47% and 38% higher than baseline.

Fig. 6 P-chart showing the percentage of hospitalized patients identified as symptomatic for COVID-19 who had COVID precautions ordered within 48 hours of PCR order in blue. The mean rate is shown in teal with upper and lower control limits (±3 SD from centerline) in red. Boxed annotations represent interventions implemented at the designated time points. COVID, coronavirus disease 2019; PCR, polymerase chain reaction; SD, standard deviation; UCL, upper control limit; LCL, lower control limit.

Our second clinical metric was the average time between when a SARS-CoV-2 PCR for a symptomatic patient was ordered and when appropriate COVID precautions were ordered. At baseline, the average time between the PCR and precautions order was 19.7 minutes but there was significant variability week-to-week as evidenced by wide control limits on our X chart and a baseline moving range of 65 minutes in our moving range (mR) chart in [Fig. 7]. There was no special cause variation observed on our X chart directly related to our initial interventions though there was a period of special cause with 11 consecutive points above the mean that began in December 2022 midway between our first intervention and our second. As this did not correlate specifically with one of our interventions, we did not shift our centerline and suspect this may have been due to increased patient volume or acuity at this time. Following replacement of the interruptive BPA with the rule-based order panel in March of 2023, we observed an immediate response in our data, with a downward shift in our mean to −1.3 minutes indicating that the precautions orders were being placed, on average, slightly ahead of the PCR orders. Additionally, there was a substantial reduction in the variability week-to-week as apparent by the narrowing of our control limits and the reduction in our moving range to 18.2 minutes.

Fig. 7 X-mR chart showing the time between SARS-CoV-2 PCR orders and COVID precautions orders among symptomatic patients. The Individuals Chart (X) is on top shows the average difference in minutes among all orders in a given week in blue, the mean of these averages in teal, and the upper and lower control limits (±3 SD from centerline) in red. The moving range (mR) chart shows the absolute difference between successive data points over time in blue, with the average range of variability in teal, and the upper and lower control limits (±3 SD from centerline) in red. Boxed annotations represent interventions implemented at the designated time points. COVID, coronavirus disease 2019; SD, standard deviation; UCL, upper control limit; LCL, lower control limit.

 

Scroll to top

 

Scroll to top

Discussion

Summary and Interpretation

We aimed in this initiative to reduce the alert burden of a single problematic BPA while supporting the clinical need for appropriate and prompt ordering of infection precautions in admitted patients with potential COVID. Through iterative design changes to better align with the “five rights” of CDS, we were able to reduce the overall alert volume and alert per encounter rate by over 80% in our health system. Furthermore, by replacing an interruptive BPA with a rule-based order panel in acute care settings, we also significantly improved the rate of appropriate infection precautions being ordered for admitted patients with potential COVID and reduced the time between a SARS-CoV-2 PCR and COVID precautions orders in these patients, which were key clinical priorities for our infection preventionists and hospital administration.

CDS has become synonymous with interruptive BPA alerts at many institutions. Often, this is due to a lack of awareness by operational stakeholders requesting CDS of the full scope of decision support options within the EHR or the assumption that a “pop up” is the most effective means available. In general, clinicians prefer “nudges” rather than interruptive alerts,[24] and passive decision support, while perhaps not entirely benign,[25] [26] does not pose the same alert fatigue risk as an interruptive alert that forcibly disrupts a clinician's workflow. While there may be high-risk circumstances necessitating an interruptive alert, it is beneficial to educate operational partners requesting a BPA about available CDS options and to take a more user-centered approach by aiming to use noninterruptive methods whenever feasible. Additionally, if interruptive CDS mechanisms are chosen, then it is critical to follow metrics on alert burden and clinical outcomes and adjust the CDS design in a data-driven fashion. Doing so provides a foundation for application of QI methodologies to not only reduce alert burden and clinician disruptions, as others have reported,[15] [19] but also, as we described here, to achieve improved clinical endpoints via iteratively designing CDS towards the “five rights.”

 

Scroll to top

Limitations

There are limitations to our work that should be acknowledged, however. First, the success of our approach was driven by institutional investment in clinical informaticists who are skilled in change management, clinical workflow analysis, and technical feasibility of CDS. This investment was recent at our institution, and we recognize that it is not universal which may limit the generalizability of this work. Additionally, our passive CDS intervention only targeted admitted patients whose SARS-CoV-2 PCR was ordered within that encounter. As some patients might have been tested prior to ED or hospital arrival, these patients would have been missed by our intervention and by our data which use provider identification of patients as symptomatic in the PCR order to identify those with symptoms. Finally, we are keenly aware that we could not control for other systems or technical forces that might have influenced the trends in our data. For example, provider ordering behavior surrounding the timing of SARS-CoV-2 PCR orders could have been altered by evolving vaccination rates and the prevalence of COVID in our community. It is notable as well that the overall alert trend did appear to rise and fall over the course of our data collection and while that was largely related to the COVID BPA trends, which appeared to be driven by our interventions, it could suggest that other factors may have been at play.

Additionally, while improvements in both our informatics and clinical metrics were clear and significant, there is still room for further improvement. The interruptive BPA remains active in the ambulatory setting and one key next step is to work on devising alternative CDS that would be less interruptive while still addressing the need to identify potential COVID patients for rapid rooming. Additionally, with 39% of symptomatic admitted patients still not having precautions orders, we clearly do have room for further optimization of our passive CDS in the acute care setting. One beneficial next step would be to carry out structured usability assessments to identify possible sources of error or poor usability that may be a factor, as this has previously been shown to be beneficial in improving decision support systems.[10] [11] Further data analysis of cases where precautions were not appropriately ordered would also be beneficial to better identify targets for optimizations or refinement to our CDS, or perhaps additional appropriate clinical exceptions that need to be incorporated.

 

Scroll to top

 

Scroll to top

Conclusion

In conclusion, we were able to successfully reduce the alert burden while simultaneously improving clinical outcomes by transitioning from an interruptive BPA to a noninterruptive rule-based order panel. Before closing, though, it is important to recognize the time and effort that was required to de-implement this single BPA in acute care settings at our institution. As with other areas of medicine where de-implementation of unnecessary or nonevidence-based practices can be particularly challenging,[27] the work to de-implement CDS is not insignificant. Our workgroup had identified this BPA as problematic from the standpoint of clinician burden well over a year before we were able to de-implement it in our inpatient, ED, and urgent care areas. Beyond the work to design and build the interventions described in this report, there were countless emails and numerous hours of meetings with operational stakeholders to achieve buy-in across our multihospital health system. This highlights the challenge of dismantling entrenched, established practices in general, but particularly when it comes to CDS within the EHR that is aimed at promoting safe care or reducing the risk of patient harm. The default assumption, even when the impact is not measured, is that a BPA, once embedded within clinical workflows, must be doing something and negative outcomes will result if it is taken away. This presumption and the time and effort required to de-implement this BPA further emphasizes the crucial importance of thoughtful user-centered design and upfront CDS governance at the time of implementation of an interruptive BPA. Only with careful governance, including a plan for prospective evaluation and planned de-implementation of ineffective alerts, will we arrive at intelligent and effective CDS that achieves intended clinical outcomes without overburdening clinicians.

 

Scroll to top

Clinical Relevance Statement

Given the risks of alert fatigue, governance, evaluation, and optimization are critical when implementing CDS in the EHR. Here, we report on how we replaced an interruptive alert with noninterruptive CDS that achieved better clinical endpoints with reduced alert burden.

 

Scroll to top

Multiple Choice Questions

1. At what stage of clinical decision support design and implementation is it important to consider a governance plan, including how to evaluate CDS efficacy, alert burden, and outcomes?

   • Following implementation

   • Prior to implementation

   • When there is negative feedback received about the alert from clinicians

   Correct answer: b. Prior to implementation. Developing a governance plan prior to implementation of a new CDS intervention, particularly when an interruptive alert, is best practice. It is helpful at the outset of a request for a new CDS tool to clearly define the clinical outcome that it is aiming to achieve as well as a plan to measure that outcome prospectively following implementation, along with metrics related to the alert burden.

2. Your institution's chief quality officer (CQO) requests a new “pop-up” to appear to ordering providers every time an expiring medication is about to fall off the patient's medication list after a patient experienced a seizure due to an antiepileptic medication auto-discontinuing. She indicates this will help to prevent this from occurring in the future. As the informaticist assigned to assist with this request, what are key considerations to discuss with the CQO?

   • Risks of alert fatigue due to a new interruptive alert

   • Alternative decision support systems that are nonpassive besides a “pop-up”

   • Plan for prospective evaluation of clinical outcome and alert burden metrics

   • All of the above.

   Correct answer: d. All of the above. It is important to partner with operational stakeholders who request new clinical decision support systems to educate them about the risks of alert fatigue and the alternatives to interruptive alerts. Collaboratively developing a plan for prospective evaluation of the efficacy following implementation is also critical to facilitate data-driven adjustments to the CDS to ensure clinical efficacy with minimal alert burden.

 

Scroll to top

 

Scroll to top

Conflict of Interest

None declared.

Acknowledgments

We acknowledge the contribution from operational partners from our infection prevention and clinical leadership teams who facilitated this effort.

Protection of Human and Animal Subjects

This project was deemed exempt as non-Human Subject Research per our Institutional Review Board. Following query validation, no protected health information was accessed and only aggregated de-identified data were stored and analyzed on an internal institutional OneDrive compliant with the Health Insurance Portability and Accountability Act (HIPAA).

• References

• 1 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 2 Osheroff JA, Teich JM, Levick D. et al. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed. Chicago, Illinois: HIMSS Publishing; 2012
• 3 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 4 Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden. Appl Clin Inform 2019; 10 (05) 909-917
• 5 McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 (02) 195-202
• 6 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
• 7 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 8 Carspecken CW, Sharek PJ, Longhurst C, Pageler NM. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics 2013; 131 (06) e1970-e1973
• 9 Westbrook JI, Woods A, Rob MI, Dunsmuir WT, Day RO. Association of interruptions with an increased risk and severity of medication administration errors. Arch Intern Med 2010; 170 (08) 683-690
• 10 Miller SD, Murphy Z, Gray JH. et al. Human-centered design of a clinical decision support for anemia screening in children with inflammatory bowel disease. Appl Clin Inform 2023; 14 (02) 345-353
• 11 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
• 12 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
• 13 Samal L, Wu E, Aaron S. et al. Refining clinical phenotypes to improve clinical decision support and reduce alert fatigue: a feasibility study. Appl Clin Inform 2023; 14 (03) 528-537
• 14 Saiyed SM, Greco PJ, Fernandes G, Kaelber DC. Optimizing drug-dose alerts using commercial software throughout an integrated health care system. J Am Med Inform Assoc 2017; 24 (06) 1149-1154
• 15 Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing interruptive alert burden using quality improvement methodology. Appl Clin Inform 2020; 11 (01) 46-58
• 16 McCoy AB, Russo EM, Johnson KB. et al. Clinician collaboration to improve clinical decision support: the Clickbusters initiative. J Am Med Inform Assoc 2022; 29 (06) 1050-1059
• 17 Chaparro JD, Beus JM, Dziorny AC. et al. Clinical decision support stewardship: best practices and techniques to monitor and improve interruptive alerts. Appl Clin Inform 2022; 13 (03) 560-568
• 18 Van Dort BA, Zheng WY, Sundar V, Baysari MT. Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals. J Am Med Inform Assoc 2021; 28 (01) 177-183
• 19 Ng HJH, Kansal A, Abdul Naseer JF. et al. Optimizing Best Practice Advisory alerts in electronic medical records with a multi-pronged strategy at a tertiary care hospital in Singapore. JAMIA Open 2023; 6 (03) ooad056
• 20 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf 2016; 25 (12) 986-992
• 21 Orenstein EW, Kandaswamy S, Muthu N. et al. Alert burden in pediatric hospitals: a cross-sectional analysis of six academic pediatric health systems using novel metrics. J Am Med Inform Assoc 2021; 28 (12) 2654-2660
• 22 Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd ed. Chichester, England: Jossey-Bass; 2009
• 23 Provost LP, Murray SK. The Health Care Data Guide: Learning from Data for Improvement. 1st ed. San Francisco, California: Jossey-Bass; 2011
• 24 Trinkley KE, Blakeslee WW, Matlock DD. et al. Clinician preferences for computerised clinical decision support for medications in primary care: a focus group study. BMJ Health Care Inform 2019;26(01):0
• 25 Kizzier-Carnahan V, Artis KA, Mohan V, Gold JA. Frequency of passive EHR alerts in the ICU: another form of alert fatigue?. J Patient Saf 2019; 15 (03) 246-250
• 26 Nijor S, Rallis G, Lad N, Gokcen E. Patient safety issues from information overload in electronic medical records. J Patient Saf 2022; 18 (06) e999-e1003
• 27 Montini T, Graham ID. “Entrenched practices and other biases”: unpacking the historical, economic, professional, and social resistance to de-implementation. Implement Sci 2015; 10 (01) 24

Publication History

Received: 24 September 2023

Accepted: 11 December 2023

Accepted Manuscript online:
12 December 2023

Article published online:
31 January 2024