Assessing Medication CDS Usability: Pilot Results from 10 Outpatient Clinics

Abstract

Objectives

This study aimed to develop a human factors assessment for medication-related clinical decision support (CDS) based on a previously validated tool that assessed the integration of human factors principles in CDS, the instrument for evaluating human factors principles in medication-related decision support alerts (I-MeDeSA), and pilot it with 10 outpatient clinics across the United States.

Methods

The human factors assessment was developed based on past validations of I-MeDeSA. Examples included changing the wording of questions and reformatting answer choices to check-box options, allowing for multiple answer choices. We also added a section about how clinicians resolved alerts. Clinics received a percentage score based on how well their CDS adhered to human factors principles. To take the assessment, testing teams at each clinic triggered a high-severity drug–drug interaction (DDI) alert, and then took the human factors assessment. This assessment was piloted in 10 outpatient clinics, each of which used a different commercial electronic health record (EHR) system.

Results

The final assessment included five sections and twelve questions related to aspects like the timing, visual aspect, severity, content, and actions within the DDI alert. The mean overall percentage score was 62%. The sections regarding the timing and visual aspects of the alert were ones where clinics' EHRs performed the best. However, in the “actions” section, 40% of the clinics could bypass high severity alerts without any safeguards in place.

Conclusion

We found substantial variability in the integration of human factors principles in the design and delivery of DDI alerts among the outpatient clinics, and some lacked important medication safeguards. This assessment can be used by outpatient clinics for safety improvement initiatives.

Keywords

medication safety - quality of care - psychological burnout - ambulatory care - electronic health record - human factors - usability - artificial intelligence

Protection of Human and Animal Subjects

No human subjects were involved in this study, as all testing scenarios involved fictitious patient scenarios.

Publication History

Received: 28 January 2025

Accepted: 01 July 2025

Article published online:
20 August 2025

• References

• 1 Office of the National Coordinator for Health Information Technology. Office-based Physician Electronic Health Record Adoption. Health IT Quick-Stat #50. January 2019. Accessed August 4, 2021 at: https://dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
• 2 Classen DC, Holmgren AJ, Co Z. et al. National trends in the safety performance of electronic health record systems from 2009 to 2018. JAMA Netw Open 2020; 3 (05) e205547
• 3 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med 2010; 363 (06) 501-504
• 4 Bates DW, Teich JM, Lee J. et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6 (04) 313-321
• 5 Austin JA, Barras MA, Woods LS, Sullivan CM. The effect of digitization on the safe management of anticoagulants. Appl Clin Inform 2022; 13 (04) 845-856
• 6 Troiano D, Jones MA, Smith AH. et al; American Society of Health-System Pharmacists. ASHP guidelines on the design of database-driven clinical decision support: strategic directions for drug database and electronic health records vendors. Am J Health Syst Pharm 2015; 72 (17) 1499-1505
• 7 Phansalkar S, Edworthy J, Hellier E. et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17 (05) 493-501
• 8 Aufegger L, Serou N, Chen S, Franklin BD. Evaluating users' experiences of electronic prescribing systems in relation to patient safety: a mixed methods study. BMC Med Inform Decis Mak 2020; 20 (01) 62
• 9 Carayon P, Hoonakker P. Human factors and usability for health information technology: old and new challenges. Yearb Med Inform 2019; 28 (01) 71-77
• 10 Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45 (06) 1202-1216
• 11 Alert Fatigue. Alert Fatigue. Published online September 7, 2019. Accessed December 12, 2024 at: https://psnet.ahrq.gov/primer/alert-fatigue
• 12 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 13 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609
• 14 Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc 2014; 21 (e2): e332-e340
• 15 Adams KT, Pruitt Z, Kazi S. et al. Identifying health information technology usability issues contributing to medication errors across medication process stages. J Patient Saf 2021; 17 (08) e988-e994
• 16 Zachariah M, Phansalkar S, Seidling HM. et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems–I-MeDeSA. J Am Med Inform Assoc 2011; 18 (Suppl. 01) i62-i72
• 17 Baysari MT, Lowenstein D, Zheng WY, Day RO. Reliability, ease of use and usefulness of I-MeDeSA for evaluating drug-drug interaction alerts in an Australian context. BMC Med Inform Decis Mak 2018; 18 (01) 83
• 18 Cho I, Lee J, Han H, Phansalkar S, Bates DW. Evaluation of a Korean version of a tool for assessing the incorporation of human factors into a medication-related decision support system: the I-MeDeSA. Appl Clin Inform 2014; 5 (02) 571-588
• 19 Nanji KC, Seger DL, Slight SP. et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (05) 476-481
• 20 Co Z, Holmgren AJ, Classen DC. et al. The development and piloting of the ambulatory electronic health record evaluation tool: lessons learned. Appl Clin Inform 2021; 12 (01) 153-163
• 21 Co Z, Classen DC, Cole JM. et al. How safe are outpatient electronic health records? An evaluation of medication-related decision support using the ambulatory electronic health record evaluation tool. Appl Clin Inform 2023; 14 (05) 981-991
• 22 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 23 Harris PA, Taylor R, Minor BL. et al; REDCap Consortium. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019; 95: 103208
• 24 Zheng WY, Van Dort B, Marcilly R. et al. A tool for evaluating medication alerting systems: development and initial assessment. JMIR Med Inform 2021; 9 (07) e24022
• 25 Pruitt Z, Howe JL, Krevat SA, Khairat S, Ratwani RM. Development and pilot evaluation of an electronic health record usability and safety self-assessment tool. JAMIA Open 2022; 5 (03) ooac070
• 26 Payne TH, Hines LE, Chan RC. et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (06) 1243-1250
• 27 McGreevey III JD, Mallozzi CP, Perkins RM, Shelov E, Schreiber R. Reducing alert burden in electronic health records: state of the art recommendations from four health systems. Appl Clin Inform 2020; 11 (01) 1-12
• 28 Liu S, Kawamoto K, Del Fiol G. et al. The potential for leveraging machine learning to filter medication alerts. J Am Med Inform Assoc 2022; 29 (05) 891-899
• 29 Seidling HM, Phansalkar S, Seger DL. et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011; 18 (04) 479-484
• 30 Cahill M, Cleary BJ, Cullinan S. The influence of electronic health record design on usability and medication safety: systematic review. BMC Health Serv Res 2025; 25 (01) 31
• 31 Justice II CM, Nevin C, Neely RL. et al. Effect of tiered implementation of clinical decision support system for acute kidney injury and nephrotoxin exposure in cardiac surgery patients. Appl Clin Inform 2025; 16 (01) 1-10
• 32 Institute of Safe Medication Practices. FDA and ISMP Lists of Look-Alike Drug Names with Recommended Tall Man (Mixed Case) Letters. 2023. Accessed June 2, 2025 at: https://psnet.ahrq.gov/issue/fda-and-ismp-lists-look-alike-drug-names-recommended-tall-man-letters
• 33 Waterson J, Al-Jaber R, Kassab T, Al-Jazairi AS. Twelve-month review of infusion pump near-miss medication and dose selection errors and user-initiated “good save” corrections: retrospective study. JMIR Hum Factors 2020; 7 (03) e20364
• 34 Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007; 2007: 26-30
• 35 Samal L, Chen SW, Lipsitz S. et al. User actions within a clinical decision support alert for the management of hypertension in chronic kidney disease. Appl Clin Inform 2025; 16 (03) 595-603
• 36 Ratwani RM, Savage E, Will A. et al. Identifying electronic health record usability and safety challenges in pediatric settings. Health Aff (Millwood) 2018; 37 (11) 1752-1759
• 37 Classen DC, Longhurst CA, Davis T, Milstein JA, Bates DW. Inpatient EHR user experience and hospital EHR safety performance. JAMA Netw Open 2023; 6 (09) e2333152-e2333152
• 38 Marmor RA, Clay B, Millen M, Savides TJ, Longhurst CA. The impact of physician EHR usage on patient satisfaction. Appl Clin Inform 2018; 9 (01) 11-14
• 39 Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-Zéphir MC. Usability flaws in medication alerting systems: impact on usage and work system. Yearb Med Inform 2015; 10 (01) 55-67
• 40 Rizvi RF, Marquard JL, Hultman GM, Adam TJ, Harder KA, Melton GB. Usability evaluation of electronic health record system around clinical notes usage-an ethnographic study. Appl Clin Inform 2017; 8 (04) 1095-1105
• 41 Alami J, Hammonds C, Hensien E. et al. Usability challenges with electronic health records (EHRs) during prerounding on pediatric inpatients. JAMIA Open 2022; 5 (01) ooac018