Building Usability Knowledge for Health Information Technology: A Usability-Oriented Analysis of Incident Reports

Abstract

Background The contribution of usability flaws to patient safety issues is acknowledged but not well-investigated. Free-text descriptions of incident reports may provide useful data to identify the connection between health information technology (HIT) usability flaws and patient safety.

Objectives This article examines the feasibility of using incident reports about HIT to learn about the usability flaws that affect patient safety. We posed three questions: (1) To what extent can we gain knowledge about usability issues from incident reports? (2) What types of usability flaws, related usage problems, and negative outcomes are reported in incidents reports? (3) What are the reported usability issues that give rise to patient safety issues?

Methods A sample of 359 reports from the U.S. Food and Drug Administration Manufacturer and User Facility Device Experience database was examined. Descriptions of usability flaws, usage problems, and negative outcomes were extracted and categorized. A supplementary analysis was performed on the incidents which contained the full chain going from a usability flaw up to a patient safety issue to identify the usability issues that gave rise to patient safety incidents.

Results A total of 249 reports were included. We found that incident reports can provide knowledge about usability flaws, usage problems, and negative outcomes. Thirty-six incidents report how usability flaws affected patient safety (ranging from incidents without consequence, to death) involving electronic patient scales, imaging systems, and HIT for medication management. The most significant class of involved usability flaws is related to the reliability, the understandability, and the availability of the clinical information.

Conclusion Incidents reports involving HIT are an exploitable source of information to learn about usability flaws and their effects on patient safety. Results can be used to convince all stakeholders involved in the HIT system lifecycle that usability should be considered seriously to prevent patient safety incidents.

Keywords

information technology - patient safety - ergonomics - databases, factual - software

Protection of Human and Animal Subjects

Human and/or animal subjects were not included in the project.

• References

• 1 Chaudhry B, Wang J, Wu S. , et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752
• 2 Shekelle PG, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology. Rockville, MD: Agency for Healthcare Research and Quality; 2006
• 3 Kaplan B, Harris-Salamone KD. Health IT success and failure: recommendations from literature and an AMIA workshop. J Am Med Inform Assoc 2009; 16 (03) 291-299
• 4 Nolan ME, Siwani R, Helmi H, Pickering BW, Moreno-Franco P, Herasevich V. Health IT usability focus section: data use and navigation patterns among medical ICU clinicians during electronic chart review. Appl Clin Inform 2017; 8 (04) 1117-1126
• 5 Kim MO, Coiera E, Magrabi F. Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review. J Am Med Inform Assoc 2017; 24 (02) 246-250
• 6 US Joint Commission on Accreditation of Healthcare Organizations. Sentinel Event Alert 2016;54. Available at: https://www.jointcommission.org/assets/1/6/SEA_54_HIT_4_26_16.pdf . Accessed February 21, 2019
• 7 Ratwani R, Fairbanks T, Savage E. , et al. Mind the Gap. A systematic review to identify usability and safety challenges and practices during electronic health record implementation. Appl Clin Inform 2016; 7 (04) 1069-1087
• 8 Lavery D, Cockton G, Atkinson MP. Comparison of evaluation methods using structured usability problem reports. Behav Inf Technol 1997; 16 (4–5): 246-266
• 9 Marcilly R, Ammenwerth E, Vasseur F, Roehrer E, Beuscart-Zéphir M-C. Usability flaws of medication-related alerting functions: a systematic qualitative review. J Biomed Inform 2015; 55: 260-271
• 10 Marcilly R, Ammenwerth E, Roehrer E, Niès J, Beuscart-Zéphir M-C. Evidence-based usability design principles for medication alerting systems. BMC Med Inform Decis Mak 2018; 18 (01) 69
• 11 Scapin DL, Bastien JMC. Ergonomic criteria for evaluating the ergonomic quality of interactive systems. Behav Inf Technol 1997; 16 (4–5): 220-231
• 12 Lewis C, Wharton C. Cognitive walkthroughs. In: Helander M, Landauer TK, Prabhu P. , eds. Handbook of Human-Computer Interaction. 2nd ed. Amsterdam: Elsevier Science; 1997: 717-732
• 13 Masci P, Zhang Y, Jones P, Thimbleby H, Curzon P. A Generic User Interface Architecture for Analyzing Use Hazards in Infusion Pump Software. Schloss Dagstuhl - Leibniz-Zent Fuer Inform GmbH WadernSaarbruecken Ger; 2014
• 14 Watbled L, Marcilly R, Guerlinger S, Bastien JC, Beuscart-Zéphir M-C, Beuscart R. Combining usability evaluations to highlight the chain that leads from usability flaws to usage problems and then negative outcomes. J Biomed Inform 2018; 78: 12-23
• 15 Bastien JMC. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform 2010; 79 (04) e18-e23
• 16 Peute LWP, de Keizer NF, Jaspers MWM. The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool. J Biomed Inform 2015; 55: 1-10
• 17 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112
• 18 Koppel R, Metlay JP, Cohen A. , et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203
• 19 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (Suppl. 01) S21-S27
• 20 Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. Prescribers' interactions with medication alerts at the point of prescribing: a multi-method, in situ investigation of the human-computer interaction. Int J Med Inform 2012; 81 (04) 232-243
• 21 Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Inform Assoc 2005; 12 (04) 438-447
• 22 Waring JJ. Beyond blame: cultural barriers to medical incident reporting. Soc Sci Med 2005; 60 (09) 1927-1935
• 23 Lewis A, Williams JG. Inefficient clinical incident reporting systems create problems in learning from errors. In: Ascona, Switzerland; 2013. . Available at: http://www.chi-med.ac.uk/publicdocs/WP171.pdf . Accessed February 26, 2019
• 24 Myketiak C, Concannon S, Curzon P. Narrative perspective, person references, and evidentiality in clinical incident reports. J Pragmatics 2017; 117: 139-154
• 25 Magrabi F, Ong M-S, Runciman W, Coiera E. Using FDA reports to inform a classification for health information technology safety problems. J Am Med Inform Assoc 2012; 19 (01) 45-53
• 26 Fong A, Adams KT, Gaunt MJ, Howe JL, Kellogg KM, Ratwani RM. Identifying health information technology related safety event reports from patient safety event report databases. J Biomed Inform 2018; 86: 135-142
• 27 Magrabi F, Ong M-S, Runciman W, Coiera E. An analysis of computer-related patient safety incidents to inform the development of a classification. J Am Med Inform Assoc 2010; 17 (06) 663-670
• 28 Magrabi F, Liaw ST, Arachi D, Runciman W, Coiera E, Kidd MR. Identifying patient safety problems associated with information technology in general practice: an analysis of incident reports. BMJ Qual Saf 2016; 25 (11) 870-880
• 29 Warm D, Edwards P. Classifying health information technology patient safety related incidents - an approach used in Wales. Appl Clin Inform 2012; 3 (02) 248-257
• 30 Samaranayake NR, Cheung STD, Chui WCM, Cheung BMY. Technology-related medication errors in a tertiary hospital: a 5-year analysis of reported medication incidents. Int J Med Inform 2012; 81 (12) 828-833
• 31 Lyons I, Blandford A. Safer healthcare at home: detecting, correcting and learning from incidents involving infusion devices. Appl Ergon 2018; 67: 104-114
• 32 Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-Zéphir M-C. Usability flaws in medication alerting systems: impact on usage and work system. Yearb Med Inform 2015; 10 (01) 55-67
• 33 IEC 62366–1:2015(en,fr), Medical devices — Part 1: Application of usability engineering to medical devices / Dispositifs médicaux — Partie 1: Application de l'ingénierie de l'aptitude à l'utilisation aux dispositifs médicaux. Available at: https://www.iso.org/obp/ui/#iso:std:iec:62366:-1:ed-1:v1:en,fr . Accessed September 20, 2018
• 34 Magrabi F, Baker M, Sinha I. , et al. Clinical safety of England's national programme for IT: a retrospective analysis of all reported safety events 2005 to 2011. Int J Med Inform 2015; 84 (03) 198-206
• 35 Runciman W, Hibbert P, Thomson R, Van Der Schaaf T, Sherman H, Lewalle P. Towards an International Classification for Patient Safety: key concepts and terms. Int J Qual Health Care 2009; 21 (01) 18-26
• 36 Khajouei R, Jaspers MWM. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med 2010; 49 (01) 3-19 10.3414/ME0630
• 37 Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57 (05) 805-834
• 38 Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015; 39 (02) 1
• 39 FDA. Changes to Existing Medical Software Policies Resulting from Section 3060 of the 21st Century Cures Act - Draft Guidance for Industry and Food and Drug Administration Staff. December 2017 Available at: https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM587820.pdf . Accessed February 25, 2019
• 40 Runciman WB, Webb RK, Lee R, Holland R. The Australian Incident Monitoring Study. System failure: an analysis of 2000 incident reports. Anaesth Intensive Care 1993; 21 (05) 684-695
• 41 Makeham MA, Stromer S, Bridges-Webb C. , et al. Patient safety events reported in general practice: a taxonomy. Qual Saf Health Care 2008; 17 (01) 53-57
• 42 CHI + MED - Incidents and investigation. Available at: http://www.chi-med.ac.uk/incidents/index.php . Accessed February 26, 2019
• 43 Yusop NSM, Grundy J, Vasa R. Reporting usability defects: a systematic literature review. IEEE Trans Softw Eng 2017; 43 (09) 848-867
• 44 Chai KEK, Anthony S, Coiera E, Magrabi F. Using statistical text classification to identify health information technology incidents. J Am Med Inform Assoc 2013; 20 (05) 980-985
• 45 Richards L. Using NVivo in Qualitative Research. London: Sage Publications; 1999
• 46 Hixenbaugh AF. Fault Tree for Safety. Seattle, WA: The Boeing Company; 1968