Appl Clin Inform 2019; 10(05): 927-934
DOI: 10.1055/s-0039-3400447
Research Article
Georg Thieme Verlag KG Stuttgart · New York

The Effect of Eliminating Intermediate Severity Drug-Drug Interaction Alerts on Overall Medication Alert Burden and Acceptance Rate

Amy M. Knight
1   Division of Hospital Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, United States
Joyce Maygers
2   Department of Care Management, Johns Hopkins Bayview Medical Center, Baltimore, Maryland, United States
Kimberly A. Foltz
3   Division of Clinical Informatics, Department of Information Services, Johns Hopkins Bayview Medical Center, Baltimore, Maryland, United States
Isha S. John
4   American Pharmacists Association, Washington, District of Columbia, United States
Hsin Chieh Yeh
5   Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States
6   Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, United States
Daniel J. Brotman
6   Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, United States
› Author Affiliations
Funding This publication was made possible by the Johns Hopkins Institute for Clinical and Translational Research (ICTR) which is funded in part by Grant Number UL1 TR 001079 from the National Center for Advancing Translational Sciences (NCATS) a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the Johns Hopkins ICTR, NCATS or NIH.
Further Information

Publication History

12 March 2019

07 October 2019

Publication Date:
04 December 2019 (online)


Objective This study aimed to determine the effects of reducing the number of drug-drug interaction (DDI) alerts in an order entry system.

Methods Retrospective pre–post analysis at an urban medical center of the rates of medication alerts and alert acceptance during a 5-month period before and 5-month period after the threshold for firing DDI alerts was changed from “intermediate” to “severe.” To ensure that we could determine varying response to each alert type, we took an in-depth look at orders generating single alerts.

Results Before the intervention, 241,915 medication orders were placed, of which 25.6% generated one or more medication alerts; 5.3% of the alerts were accepted. During the postintervention period, 245,757 medication orders were placed of which 16.0% generated one or more medication alerts, a 37.5% relative decrease in alert rate (95% confidence interval [CI]: −38.4 to −36.8%), but only a 9.6% absolute decrease (95% CI: −9.4 to −9.9%). 7.4% of orders generating alerts were accepted postintervention, a 39.6% relative increase in acceptance rate (95% CI: 33.2–47.2%), but only a 2.1% absolute increase (95% CI: 1.8–2.4%). When only orders generating a single medication alert were considered, there was a 69.1% relative decrease in the number of orders generating DDI alerts, and an 85.7% relative increase in the acceptance rate (95% CI: 58.6–126.2%), though only a 1.8% absolute increase (95% CI: 1.3–2.3%).

Conclusion Eliminating intermediate severity DDI alerts resulted in a statistically significant decrease in alert burden and increase in the rate of medication alert acceptance, but alert acceptance remained low overall.

Authors' Contributions

All listed authors contributed substantially to the study conception and design or analysis and interpretation of data, drafting the article, or revising it critically for important intellectual content, and final approval of the version to be published. No one who fulfills these criteria has been excluded from authorship.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Patients, and was reviewed by Johns Hopkins Institutional Review Board.

  • References

  • 1 Shortliffe EH. Computer programs to support clinical decision making. JAMA 1987; 258 (01) 61-66
  • 2 Clinical decision support. Available at: . Accessed August 13, 2019
  • 3 Bright TJ, Wong A, Dhurjati R. , et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
  • 4 Nuckols TK, Smith-Spangler C, Morton SC. , et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev 2014; 3: 56
  • 5 Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 2012; 125 (02) 209.e1-209.e7
  • 6 Weingart SN, Simchowitz B, Shiman L. , et al. Clinicians' assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med 2009; 169 (17) 1627-1632
  • 7 Glassman PA, Simon B, Belperio P, Lanto A. Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care 2002; 40 (12) 1161-1171
  • 8 Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006; 13 (05) 547-556
  • 9 Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014; 23 (09) 773-780
  • 10 Kane-Gill SL, O'Connor MF, Rothschild JM. , et al. Technologic distractions (part 1): Summary of approaches to manage alert quantity with intent to reduce alert fatigue and suggestions for alert fatigue metrics. Crit Care Med 2017; 45 (09) 1481-1488
  • 11 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
  • 12 Isaac T, Weissman JS, Davis RB. , et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169 (03) 305-311
  • 13 Nanji KC, Slight SP, Seger DL. , et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21 (03) 487-491
  • 14 Carspecken CW, Sharek PJ, Longhurst C, Pageler NM. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics 2013; 131 (06) e1970-e1973
  • 15 Wachter R. The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine's Computer Age. 1st ed. New York, NY: McGraw Hill Education; 2015
  • 16 Wong A, Rehr C, Seger DL. , et al. Evaluation of harm associated with high dose-range clinical decision support overrides in the intensive care unit. Drug Saf 2019; 42 (04) 573-579
  • 17 Knight AM, Falade O, Maygers J, Sevransky JE. Factors associated with medication warning acceptance for hospitalized adults. J Hosp Med 2015; 10 (01) 19-25
  • 18 Thiemann DR. Chair, Johns Hopkins Hospital Clinical Decision Support Committee. Personal communication, 10/14/2013.
  • 19 McAlduff J. , VP-CMIO, MedStar Health, Columbia, Maryland. Email communication, 11/27–28/13
  • 20 Institute for Safe Medication Practice. High-alert medications in acute care setting. Available at: . Accessed August 5, 2019
  • 21 Payne TH, Hines LE, Chan RC. , et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (06) 1243-1250
  • 22 Phansalkar S, van der Sijs H, Tucker AD. , et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 (03) 489-493
  • 23 Bates DW. CPOE and clinical decision support in hospitals: getting the benefits: comment on “Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction”. Arch Intern Med 2010; 170 (17) 1583-1584
  • 24 Phansalkar S, Edworthy J, Hellier E. , et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17 (05) 493-501
  • 25 Classen DC, Phansalkar S, Bates DW. Critical drug-drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf 2011; 7 (02) 61-65
  • 26 van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008; 15 (04) 439-448
  • 27 van der Sijs H, Kowlesar R, Aarts J, Berg M, Vulto A, van Gelder T. Unintended consequences of reducing QT-alert overload in a computerized physician order entry system. Eur J Clin Pharmacol 2009; 65 (09) 919-925
  • 28 Lee EK, Mejia AF, Senior T, Jose J. Improving patient safety through medical alert management: An automated decision tool to reduce alert fatigue. AMIA Annu Symp Proc 2010; 2010: 417-421
  • 29 Resetar E, Reichley RM, Noirot LA, Doherty JA, Dunagan WC, Bailey TC. Strategies for reducing nuisance alerts in a dose checking application. AMIA Annu Symp Proc 2005; 624-628
  • 30 Beccaro MA, Villanueva R, Knudson KM, Harvey EM, Langle JM, Paul W. Decision support alerts for medication ordering in a computerized provider order entry (CPOE) system: A systematic approach to decrease alerts. Appl Clin Inform 2010; 1 (03) 346-362
  • 31 Schreiber R, Gregoire JA, Shaha JE, Shaha SH. Think time: a novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts. Int J Med Inform 2017; 97: 59-67
  • 32 Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inform 2014; 5 (03) 802-813
  • 33 Dexheimer JW, Kirkendall ES, Kouril M. , et al. The effects of medication alerts on prescriber response in a pediatric hospital. Appl Clin Inform 2017; 8 (02) 491-501
  • 34 Schiff GD, Hickman TT, Volk LA, Bates DW, Wright A. Computerised prescribing for safer medication ordering: still a work in progress. BMJ Qual Saf 2016; 25 (05) 315-319
  • 35 Wright A, Sittig DF, Ash JS. , et al. Governance for clinical decision support: case studies and recommended practices from leading institutions. J Am Med Inform Assoc 2011; 18 (02) 187-194
  • 36 Kawamanto K, Flynn MC, Kukhareva P. , et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: A case study. AMIA Annu Symp Proc 2018; 2018: 624-633
  • 37 Rehr CA, Wong A, Seger DL, Bates DW. Determining inappropriate medication alerts from “inaccurate warning” overrides in the intensive care unit. Appl Clin Inform 2018; 9 (02) 268-274
  • 38 Ko Y, Abarca J, Malone DC. , et al. Practitioners' views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007; 14 (01) 56-64
  • 39 Spina JR, Glassman PA, Simon B. , et al. Potential safety gaps in order entry and automated drug alerts: a nationwide survey of VA physician self-reported practices with computerized order entry. Med Care 2011; 49 (10) 904-910
  • 40 Beeler PE, Orav EJ, Seger DL, Dykes PC, Bates DW. Provider variation in responses to warnings: do the same providers run stop signs repeatedly?. J Am Med Inform Assoc 2016; 23 (e1): e93-e98
  • 41 Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce ‘alert fatigue’ while still minimizing the risk of litigation. Health Aff (Millwood) 2011; 30 (12) 2310-2317
  • 42 Phansalkar S, Desai AA, Bell D. , et al. High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc 2012; 19 (05) 735-743
  • 43 McEvoy DS, Sittig DF, Hickman TT. , et al. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017; 24 (02) 331-338
  • 44 Tilson H, Hines LE, McEvoy G. , et al. Recommendations for selecting drug-drug interactions for clinical decision support. Am J Health Syst Pharm 2016; 73 (08) 576-585
  • 45 Nanji KC, Seger DL, Slight SP. , et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (05) 476-481
  • 46 Wong A, Amato MG, Seger DL. , et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. BMJ Qual Saf 2018; 27 (09) 718-724
  • 47 Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc 2014; 21 (e2): e332-e340
  • 48 Wong A, Amato MG, Seger DL. , et al. Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care 2017; 39: 156-161
  • 49 Coleman JJ, van der Sijs H, Haefeli WE. , et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013; 13: 111
  • 50 Paterno MD, Maviglia SM, Gorman PN. , et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 (01) 40-46
  • 51 Shah NR, Seger AC, Seger DL. , et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006; 13 (01) 5-11