Appl Clin Inform 2022; 13(02): 468-485
DOI: 10.1055/s-0042-1748146
Review Article

Modulators Influencing Medication Alert Acceptance: An Explorative Review

Janina A. Bittmann
1   Cooperation Unit Clinical Pharmacy, Heidelberg University, Heidelberg, Germany
2   Department of Clinical Pharmacology and Pharmacoepidemiology, Heidelberg University Hospital, Heidelberg, Germany
Walter E. Haefeli
1   Cooperation Unit Clinical Pharmacy, Heidelberg University, Heidelberg, Germany
2   Department of Clinical Pharmacology and Pharmacoepidemiology, Heidelberg University Hospital, Heidelberg, Germany
Hanna M. Seidling
1   Cooperation Unit Clinical Pharmacy, Heidelberg University, Heidelberg, Germany
2   Department of Clinical Pharmacology and Pharmacoepidemiology, Heidelberg University Hospital, Heidelberg, Germany
› Author Affiliations
Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.


Objectives Clinical decision support systems (CDSSs) use alerts to enhance medication safety and reduce medication error rates. A major challenge of medication alerts is their low acceptance rate, limiting their potential benefit. A structured overview about modulators influencing alert acceptance is lacking. Therefore, we aimed to review and compile qualitative and quantitative modulators of alert acceptance and organize them in a comprehensive model.

Methods In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline, a literature search in PubMed was started in February 2018 and continued until October 2021. From all included articles, qualitative and quantitative parameters and their impact on alert acceptance were extracted. Related parameters were then grouped into factors, allocated to superordinate determinants, and subsequently further allocated into five categories that were already known to influence alert acceptance.

Results Out of 539 articles, 60 were included. A total of 391 single parameters were extracted (e.g., patients' comorbidity) and grouped into 75 factors (e.g., comorbidity), and 25 determinants (e.g., complexity) were consequently assigned to the predefined five categories, i.e., CDSS, care provider, patient, setting, and involved drug. More than half of all factors were qualitatively assessed (n = 21) or quantitatively inconclusive (n = 19). Furthermore, 33 quantitative factors clearly influenced alert acceptance (positive correlation: e.g., alert type, patients' comorbidity; negative correlation: e.g., number of alerts per care provider, moment of alert display in the workflow). Two factors (alert frequency, laboratory value) showed contradictory effects, meaning that acceptance was significantly influenced both positively and negatively by these factors, depending on the study. Interventional studies have been performed for only 12 factors while all other factors were evaluated descriptively.

Conclusion This review compiles modulators of alert acceptance distinguished by being studied quantitatively or qualitatively and indicates their effect magnitude whenever possible. Additionally, it describes how further research should be designed to comprehensively quantify the effect of alert modulators.

Protection of Human and Animal Subjects

No human and/or animal subjects were involved in this study.

Supplementary Material

Publication History

Received: 09 November 2021

Accepted: 04 March 2022

Article published online:
18 August 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Zachariah M, Phansalkar S, Seidling HM. et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems–I-MeDeSA. J Am Med Inform Assoc 2011; 18 (Suppl. 01) i62-i72
  • 2 Varghese J, Kleine M, Gessner SI, Sandmann S, Dugas M. Effects of computerized decision support system implementations on patient outcomes in inpatient care: a systematic review. J Am Med Inform Assoc 2018; 25 (05) 593-602
  • 3 Olufisayo O, Mohd Yusof M, Ezat Wan Puteh S. Enhancing CDSS alert appropriateness in clinical workflow using the Lean method. Stud Health Technol Inform 2018; 255: 112-116
  • 4 Carroll AE. Averting alert fatigue to prevent adverse drug reactions. JAMA 2019; 322 (07) 601
  • 5 Lee JD, See KA. Trust in automation: designing for appropriate reliance. Hum Factors 2004; 46 (01) 50-80
  • 6 Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014; 23 (09) 773-780
  • 7 Nuckols TK, Smith-Spangler C, Morton SC. et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev 2014; 3: 56
  • 8 Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc 2011; 18 (03) 327-334
  • 9 Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12) 1409-1416
  • 10 Salili AR, Hammann F, Taegtmeyer AB. Preventing adverse drug events using clinical decision support systems [in German]. Ther Umsch 2015; 72 (11–12): 693-700
  • 11 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
  • 12 Seidling HM, Klein U, Schaier M. et al. What, if all alerts were specific - estimating the potential impact on drug interaction alert burden. Int J Med Inform 2014; 83 (04) 285-291
  • 13 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
  • 14 Shah SN, Amato MG, Garlo KG, Seger DL, Bates DW. Renal medication-related clinical decision support (CDS) alerts and overrides in the inpatient setting following implementation of a commercial electronic health record: implications for designing more effective alerts. J Am Med Inform Assoc 2021; 28 (06) 1081-1087
  • 15 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631
  • 16 Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002; 602-606
  • 17 Coleman JJ, van der Sijs H, Haefeli WE. et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013; 13: 111
  • 18 Czock D, Konias M, Seidling HM. et al. Tailoring of alerts substantially reduces the alert burden in computerized clinical decision support for drugs that should be avoided in patients with renal disease. J Am Med Inform Assoc 2015; 22 (04) 881-887
  • 19 Campbell R. The five “rights” of clinical decision support. J AHIMA 2013; 84 (10) 42-47 , quiz 48
  • 20 Berner ES. Clinical decision support systems: State of the Art. AHRQ Publication No 09–0069-EF. 2009. Rockville, MD: Agency for Healthcare Research and Quality;
  • 21 Haefeli WE, Seidling HM. Electronic decision support to promote medication safety [in German]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz 2018; 61 (03) 271-277
  • 22 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
  • 23 Van Dort BA, Zheng WY, Sundar V, Baysari MT. Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals. J Am Med Inform Assoc 2021; 28 (01) 177-183
  • 24 Miller K, Mosby D, Capan M. et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018; 25 (05) 585-592
  • 25 Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6 (07) e1000097
  • 26 Keasberry J, Scott IA, Sullivan C, Staib A, Ashby R. Going digital: a narrative overview of the clinical and organisational impacts of eHealth technologies in hospital practice. Aust Health Rev 2017; 41 (06) 646-664
  • 27 Villaseñor S, Piscotty Jr RJ. The current state of e-prescribing: implications for advanced practice registered nurses. J Am Assoc Nurse Pract 2016; 28 (01) 54-61
  • 28 Seidling HM, Bates DW. Evaluating the impact of health IT on medication safety. Stud Health Technol Inform 2016; 222: 195-205
  • 29 Rolnick J, Downing NL, Shepard J. et al. Validation of test performance and clinical time zero for an electronic health record embedded severe sepsis alert. Appl Clin Inform 2016; 7 (02) 560-572
  • 30 McClure C, Jang SY, Fairchild K. Alarms, oxygen saturations, and SpO2 averaging time in the NICU. J Neonatal Perinatal Med 2016; 9 (04) 357-362
  • 31 Chou E, Boyce RD, Balkan B. et al. Designing and evaluating contextualized drug-drug interaction algorithms. JAMIA Open 2021; 4 (01) b023
  • 32 Olakotan OO, Mohd Yusof M. The appropriateness of clinical decision support systems alerts in supporting clinical workflows: a systematic review. Health Informatics J 2021;27(02)14604582211007536
  • 33 Seidling HM, Phansalkar S, Seger DL. et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011; 18 (04) 479-484
  • 34 Payne TH, Hines LE, Chan RC. et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (06) 1243-1250
  • 35 Khairat S, Marc D, Crosby W, Al Sanousi A. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med Inform 2018; 6 (02) e24
  • 36 Olakotan OO, Yusof MM. Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow. J Biomed Inform 2020; 106: 103453
  • 37 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
  • 38 Olakotan O, Mohd Yusof M, Ezat Wan Puteh S. A systematic review on CDSS alert appropriateness. Stud Health Technol Inform 2020; 270: 906-910
  • 39 Zenziper Straichman Y, Kurnik D, Matok I. et al. Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform 2017; 107: 70-75
  • 40 Nanji KC, Seger DL, Slight SP. et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (05) 476-481
  • 41 Dexheimer JW, Kirkendall ES, Kouril M. et al. The effects of medication alerts on prescriber response in a pediatric hospital. Appl Clin Inform 2017; 8 (02) 491-501
  • 42 Cuéllar Monreal MJ, Reig Aguado J, Font Noguera I, Poveda Andrés JL. Reduction in alert fatigue in an assisted electronic prescribing system, through the Lean Six Sigma methodology. Farm Hosp 2017; 41 (01) 14-30
  • 43 Schreiber R, Gregoire JA, Shaha JE, Shaha SH. Think time: a novel approach to analysis of clinicians' behavior after reduction of drug-drug interaction alerts. Int J Med Inform 2017; 97: 59-67
  • 44 Brodowy B, Nguyen D. Optimization of clinical decision support through minimization of excessive drug allergy alerts. Am J Health Syst Pharm 2016; 73 (08) 526-528
  • 45 Cornu P, Steurbaut S, Gentens K, Van de Velde R, Dupont AG. Pilot evaluation of an optimized context-specific drug-drug interaction alerting system: a controlled pre-post study. Int J Med Inform 2015; 84 (09) 617-629
  • 46 Simpao AF, Ahumada LM, Desai BR. et al. Optimization of drug-drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard. J Am Med Inform Assoc 2015; 22 (02) 361-369
  • 47 Parke C, Santiago E, Zussy B, Klipa D. Reduction of clinical support warnings through recategorization of severity levels. Am J Health Syst Pharm 2015; 72 (02) 144-148
  • 48 Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inform 2014; 5 (03) 802-813
  • 49 Duke JD, Li X, Dexter P. Adherence to drug-drug interaction alerts in high-risk patients: a trial of context-enhanced alerting. J Am Med Inform Assoc 2013; 20 (03) 494-498
  • 50 Feblowitz J, Henkin S, Pang J. et al. Provider use of and attitudes towards an active clinical alert: a case study in decision support. Appl Clin Inform 2013; 4 (01) 144-152
  • 51 Miller AM, Boro MS, Korman NE, Davoren JB. Provider and pharmacist responses to warfarin drug-drug interaction alerts: a study of healthcare downstream of CPOE alerts. J Am Med Inform Assoc 2011; 18 (Suppl. 01) i45-i50
  • 52 Weingart SN, Seger AC, Feola N, Heffernan J, Schiff G, Isaac T. Electronic drug interaction alerts in ambulatory care: the value and acceptance of high-value alerts in US medical practices as assessed by an expert clinical panel. Drug Saf 2011; 34 (07) 587-593
  • 53 van der Sijs H, van Gelder T, Vulto A, Berg M, Aarts J. Understanding handling of drug safety alerts: a simulation study. Int J Med Inform 2010; 79 (05) 361-369
  • 54 Isaac T, Weissman JS, Davis RB. et al. Overrides of medication alerts in ambulatory care. Arch Intern Med 2009; 169 (03) 305-311
  • 55 Wright A, Aaron S, Seger DL, Samal L, Schiff GD, Bates DW. Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record. J Gen Intern Med 2018; 33 (11) 1868-1876
  • 56 Ramirez M, Maranon R, Fu J. et al. Primary care provider adherence to an alert for intensification of diabetes blood pressure medications before and after the addition of a “chart closure” hard stop. J Am Med Inform Assoc 2018; 25 (09) 1167-1174
  • 57 Choi KS, Lee E, Rhie SJ. Impact of pharmacists' interventions on physicians' decision of a knowledge-based renal dosage adjustment system. Int J Clin Pharm 2019; 41 (02) 424-433
  • 58 Bubp JL, Park MA, Kapusnik-Uner J. et al. Successful deployment of drug-disease interaction clinical decision support across multiple Kaiser Permanente regions. J Am Med Inform Assoc 2019; 26 (10) 905-910
  • 59 Hussain MI, Reynolds TL, Zheng K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: a systematic review. J Am Med Inform Assoc 2019; 26 (10) 1141-1149
  • 60 Muylle KM, Gentens K, Dupont AG, Cornu P. Evaluation of context-specific alerts for potassium-increasing drug-drug interactions: a pre-post study. Int J Med Inform 2020; 133: 104013
  • 61 Knight AM, Maygers J, Foltz KA, John IS, Yeh HC, Brotman DJ. The effect of eliminating intermediate severity drug-drug interaction alerts on overall medication alert burden and acceptance rate. Appl Clin Inform 2019; 10 (05) 927-934
  • 62 Yoo J, Lee J, Rhee PL. et al. Alert override patterns with a medication clinical decision support system in an academic emergency department: retrospective descriptive study. JMIR Med Inform 2020; 8 (11) e23351
  • 63 Al-Jazairi AS, AlQadheeb EK, AlShammari LK. et al. Clinical validity assessment of integrated dose range checking tool in a tertiary care hospital using an electronic health information system. Hosp Pharm 2021; 56 (02) 95-101
  • 64 Muylle KM, Gentens K, Dupont AG, Cornu P. Evaluation of an optimized context-aware clinical decision support system for drug-drug interaction screening. Int J Med Inform 2021; 148: 104393
  • 65 Bittmann JA, Rein EK, Metzner M, Haefeli WE, Seidling HM. The acceptance of interruptive medication alerts in an electronic decision support system differs between different alert types. Methods Inf Med 2021; 60 (05–06): 180-184
  • 66 Gadhiya K, Zamora E, Saiyed SM, Friedlander D, Kaelber DC. Drug alert experience and salience during medical residency at two healthcare institutions. Appl Clin Inform 2021; 12 (02) 355-361
  • 67 Joglekar NN, Patel Y, Keller MS. Evaluation of clinical decision support to reduce sedative-hypnotic prescribing in older adults. Appl Clin Inform 2021; 12 (03) 436-444
  • 68 Park HA. Health informatics in developing countries: a review of unintended consequences of IT implementations, as they affect patient safety and recommendations on how to address them. Yearb Med Inform 2016; (01) 1-2
  • 69 Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc 2014; 21 (e): e332-e340
  • 70 Strasberg HR, Chan A, Sklar SJ. Inter-rater agreement among physicians on the clinical significance of drug-drug interactions. AMIA Annu Symp Proc 2013; 2013: 1325-1328
  • 71 Eschmann E, Beeler PE, Zünd G, Blaser J. Evaluation of alerts for potassium-increasing drug-drug-interactions. Stud Health Technol Inform 2013; 192: 1056
  • 72 van der Sijs H, Baboe I, Phansalkar S. Human factors considerations for contraindication alerts. Stud Health Technol Inform 2013; 192: 132-136
  • 73 Eschmann E, Beeler PE, Kaplan V, Schneemann M, Zünd G, Blaser J. Clinical decision support for monitoring drug-drug-interactions and potassium-increasing drug combinations: need for specific alerts. Stud Health Technol Inform 2012; 180: 1200-1202
  • 74 Kane-Gill SL, O'Connor MF, Rothschild JM. et al. Technologic distractions (Part 1): summary of approaches to manage alert quantity with intent to reduce alert fatigue and suggestions for alert fatigue metrics. Crit Care Med 2017; 45 (09) 1481-1488
  • 75 Genco EK, Forster JE, Flaten H. et al. Clinically inconsequential alerts: the characteristics of opioid drug alerts and their utility in preventing adverse drug events in the emergency department. Ann Emerg Med 2016; 67 (02) 240.e3-248.e3
  • 76 Khalifa M, Zabani I. Improving utilization of clinical decision support systems by reducing alert fatigue: strategies and recommendations. Stud Health Technol Inform 2016; 226: 51-54
  • 77 Footracer KG. Alert fatigue in electronic health records. JAAPA 2015; 28 (07) 41-42
  • 78 Topaz M, Seger DL, Lai K. et al. High override rate for opioid drug-allergy interaction alerts: current trends and recommendations for future. Stud Health Technol Inform 2015; 216: 242-246
  • 79 McCoy AB, Thomas EJ, Krousel-Wood M, Sittig DF. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14 (02) 195-202
  • 80 Nanji KC, Slight SP, Seger DL. et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21 (03) 487-491
  • 81 Smithburger PL, Buckley MS, Bejian S, Burenheide K, Kane-Gill SL. A critical evaluation of clinical decision support for the detection of drug-drug interactions. Expert Opin Drug Saf 2011; 10 (06) 871-882
  • 82 Riedmann D, Jung M, Hackl WO, Stühlinger W, van der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak 2011; 11: 35
  • 83 Duke JD, Bolchini D. A successful model and visual design for creating context-aware drug-drug interaction alerts. AMIA Annu Symp Proc 2011; 2011: 339-348
  • 84 Smithburger PL, Kane-Gill SL, Benedict NJ, Falcione BA, Seybert AL. Grading the severity of drug-drug interactions in the intensive care unit: a comparison between clinician assessment and proprietary database severity rankings. Ann Pharmacother 2010; 44 (11) 1718-1724
  • 85 Lee EK, Wu TL, Senior T, Jose J. Medical alert management: a real-time adaptive decision support tool to reduce alert fatigue. AMIA Annu Symp Proc 2014; 2014: 845-854
  • 86 Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health Syst Pharm 2018; 75 (04) 239-246
  • 87 Chazard E, Beuscart JB, Rochoy M. et al. Statistically prioritized and contextualized clinical decision support systems, the future of adverse drug events prevention?. Stud Health Technol Inform 2020; 270: 683-687
  • 88 Poly TN, Islam MM, Yang HC, Li YJ. Appropriateness of overridden alerts in computerized physician order entry: systematic review. JMIR Med Inform 2020; 8 (07) e15653
  • 89 Wan PK, Satybaldy A, Huang L, Holtskog H, Nowostawski M. Reducing alert fatigue by sharing low-level alerts with patients and enhancing collaborative decision making using blockchain technology: scoping review and proposed framework (MedAlert). J Med Internet Res 2020; 22 (10) e22013
  • 90 Van Dort BA, Zheng WY, Sundar V, Baysari MT. Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals. J Am Med Inform Assoc 2021; 28 (01) 177-183
  • 91 Ford E, Edelman N, Somers L. et al. Barriers and facilitators to the adoption of electronic clinical decision support systems: a qualitative interview study with UK general practitioners. BMC Med Inform Decis Mak 2021; 21 (01) 193
  • 92 Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in the safety performance of computerized physician order entry. Health Aff (Millwood) 2010; 29 (04) 655-663
  • 93 Elias P, Peterson E, Wachter B, Ward C, Poon E, Navar AM. Evaluating the impact of interruptive alerts within a health system: use, response time, and cumulative time burden. Appl Clin Inform 2019; 10 (05) 909-917
  • 94 Todd B, Shinthia N, Nierenberg L, Mansour L, Miller M, Otero R. Impact of electronic medical record alerts on emergency physician workflow and medical management. J Emerg Med 2021; 60 (03) 390-395
  • 95 McGreevey III JD, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform 2020; 11 (01) 1-12
  • 96 Moja L, Kwag KH, Lytras T. et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health 2014; 104 (12) e12-e22
  • 97 Khajouei R, Jaspers MW. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010; 49 (01) 3-19