Improving the User Experience with Discount Site-Specific User Testing

Abstract

Objectives Poor electronic health record (EHR) usability is associated with patient safety concerns, user dissatisfaction, and provider burnout. EHR certification requires vendors to perform user testing. However, there are no such requirements for site-specific implementations. Health care organizations customize EHR implementations, potentially introducing usability problems. Site-specific usability evaluations may help to identify these concerns, and “discount” usability methods afford health systems a means of doing so even without dedicated usability specialists. This report characterizes a site-specific discount user testing program launched at an academic medical center. We describe lessons learned and highlight three of the EHR features in detail to demonstrate the impact of testing on implementation decisions and on users.

Methods Thirteen new EHR features which had already undergone heuristic evaluation and iterative design were evaluated over the course of three user test events. Each event included five to six users. Participants used think aloud technique. Measures of user efficiency, effectiveness, and satisfaction were collected. Usability concerns were characterized by the type of usability heuristic violated and by correctability.

Results Usability concerns occurred at a rate of 2.5 per feature tested. Seventy percent of the usability concerns were deemed correctable prior to implementation. The first highlighted feature was moved to production despite low single ease question (SEQ) scores which may have predicted its subsequent withdrawal from production based on post implementation feedback. Another feature was rebuilt based on usability findings, and a new version was retested and moved to production. A third feature highlights an easily correctable usability concern identified in user testing. Quantitative usability metrics generally reinforced qualitative findings.

Conclusion Simplified user testing with a limited number of participants identifies correctable usability concerns, even after heuristic evaluation. Our discount usability approach to site-specific usability has a role in implementations and may improve the usability of the EHR for the end user.

Keywords

electronic health records - user-centered design - user-computer interface

Protection of Human and Animal Subjects

The project details were reviewed by the University of Missouri Institutional Review Board who determined the project to be a quality improvement activity and not human subject research and did not require additional review.

Publication History

Received: 25 May 2022

Accepted: 21 September 2022

Article published online:
02 November 2022

• References

• 1 Office of the National Coordinator for Health Information Technology, Health IT Quick Stats. U.S. Department of Health and Human Services. Accessed August 05, 2021 at: https://dashboard.healthit.gov/quickstats/quickstats.php
• 2 International Standards Organization. Ergonomics of human-system interaction – Part 210: Human-centred design for interactive systems. International Organization for Standardization. 2019
• 3 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 4 Committee on Patient Safety and Health Information Technology, Institute of Medicine. Health IT and Patient Safety: Building Safer Systems for Better Care. National Academies Press (US). 2011
• 5 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 6 US Department of Health and Human Services. Health Information Technology: Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology, 2014 Edition; Revisions to the Permanent Certification Program for Health Information Technology. In: Federal Register, ed.; 2012
• 7 US Department of Health and Human Services. 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications. In: Federal Register, ed.; 2015
• 8 Tutty MA, Carlasare LE, Lloyd S, Sinsky CA. The complex case of EHRs: examining the factors impacting the EHR user experience. J Am Med Inform Assoc 2019; 26 (07) 673-677
• 9 Ratwani RM, Savage E, Will A. et al. A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inform Assoc 2018; 25 (09) 1197-1201
• 10 Meigs SL, Solomon M. Electronic health record use a bitter pill for many physicians. Perspect Health Inf Manag 2016; 13 (Winter): 1d
• 11 Ratwani RM, Sinsky CA, Melnick ER. Closing the Electronic Health Record Usability Gap. Bill of Health blog. June 26, 2020, 2020. Accessed February 8, 2021 at: https://blog.petrieflom.law.harvard.edu/2020/06/26/closing-the-electronic-health-record-usability-gap/
• 12 Nielsen J. Discount Usability: 20 Years. Nielsen Norman Group; 2022
• 13 Nielsen J. Usability Engineering at a Discount. Elsevier Science Publishers; 1989
• 14 Customized Common Industry Format Template for Electronic Health Record Usability Testing, (NISTIR 7742). National Institute for Standards and Technology. 2010: 1-37
• 15 TechSmith. TechSmith Support Policy. TechSmith. 2021 . Accessed August 05, 2021, at: https://support.techsmith.com/hc/en-us/articles/203732728
• 16 Sauro J, Dumas JS. Comparison of three one-question, post-task usability questionnaires. Paper presented at: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2009 Boston, MA DOI: 10.1145/1518701.1518946
• 17 Nielsen J. 10 Usability Heuristics for User Interface Design. Nielsen Norman Group. Accessed May 17, 2020 at: https://www.nngroup.com/articles/ten-usability-heuristics/
• 18 Corrao NJ, Robinson AG, Swiernik MA, Naeim A. Importance of testing for usability when selecting and implementing an electronic health or medical record system. J Oncol Pract 2010; 6 (03) 120-124
• 19 Pertiwi AAP, Fraczkowski D, Stogis SL, Lopez KD. Using heuristic evaluation to improve sepsis alert usability. Crit Care Nurs Clin North Am 2018; 30 (02) 297-309
• 20 Tarrell A, Grabenbauer L, McClay J, Windle J, Fruhling AL. Toward improved heuristic evaluation of EHRs. Health Syst (Basingstoke) 2015; 4 (02) 138-150
• 21 Beaudoin DE, Rocha RA, Tse T. Enhancing access to patient education information: a pilot usability study. AMIA Annu Symp Proc 2005; 2005: 892-892
• 22 Schaarup C, Hejlesen OK. Heuristic evaluation and thinking aloud test of a digitized questionnaire for diabetes outpatient clinics. Stud Health Technol Inform 2014; 205: 920-924
• 23 Khelifi M, Tarczy-Hornoch P, Devine EB, Pratt W. Design recommendations for pharmacogenomics clinical decision support systems. AMIA Jt Summits Transl Sci Proc 2017; 2017: 237-246
• 24 Sinabell I, Ammenwerth E. Agile, easily applicable, and useful ehealth usability evaluations: systematic review and expert-validation. Appl Clin Inform 2022; 13 (01) 67-79
• 25 Hettinger AZ, Melnick ER, Ratwani RM. Advancing electronic health record vendor usability maturity: progress and next steps. J Am Med Inform Assoc 2021; 28 (05) 1029-1031
• 26 Schade A. Write better qualitative usability tasks: top 10 mistakes to avoid. Nielsen Norman Group. Accessed February 21, 2021. 2021 at: https://www.nngroup.com/articles/better-usability-tasks/
• 27 Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inform 2018; 78: 123-133
• 28 Brooke J. SUS: A “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL. eds. Usability Evaluation in Industry. Taylor and Francis; 1996