Human Factors and Usability for Health Information Technology: Old and New Challenges

Summary

Objectives: Despite national mandates, incentives, and other programs, the design of health information technology (IT) remains problematic and usability problems continue to be reported. This paper reviews recent literature on human factors and usability of health IT, with a specific focus on research aimed at applying human factors methods and principles to improve the actual design of health IT, its use, and associated patient and clinician outcomes.

Methods: We reviewed recent literature on human factors and usability problems of health IT and research on human-centered design of health IT for clinicians and patients.

Results: Studies continue to show usability problems of health IT experienced by multiple groups of health care professionals (e.g., physicians and nurses) as well as patients. Recent research shows that usability is influenced by both designers (e.g., IT vendors) and implementers in health care organizations, and that the application of human-centered design practices needs to be further improved and extended. We welcome emerging research on the design of health IT for teams as team-based care is increasingly implemented throughout health care.

Conclusions: Progress in the application of human factors methods and principles to the design of health IT is occurring, with important information provided on their actual impact on care processes and patient outcomes. Future research should examine the work of health IT designers and implementers, which would help to develop strategies for further embedding human factors engineering in IT design processes.

 

MissingLinkText

Keywords

Human factors engineering - usability - human-centered design - health information technology - patient safety

1 Introduction

In several countries, we are moving beyond issues related to implementation and change management of health information technology (IT); today, the basic IT infrastructure is in place and health care organizations are investing in designing and implementing the next generation of health IT “by-products”, such as clinical dashboard, status display, clinical decision support, and patient-facing technologies. Usability and other human factors challenges with health IT continue to be reported, as illustrated in a New Yorker article by Atul Gawande entitled “Why doctors hate their computers” [1]. The experience of usability problems with health IT extends to other health care professionals, including nurses [2]. Staggers et al. [2] gathered input on health IT problems experienced by nurses from a group of nursing leaders, IT experts, engineers and human factors experts. The consensus among the group of 27 experts was that “current vendors do not yet understand how to support nurses’ work, their critical thinking, and decision making” (page 192). For instance, current EHR designs allow for single patient views whereas nurses care for complex groups of patients in a non-linear, non-standardized manner. Patients also experience usability challenges with health information technologies, such as patient portals [3].

Over time, implementation of health IT may lead to benefits for patients as demonstrated by the study by Lin et al. [4], which shows positive outcomes (e.g. hospital mortality) associated with health IT after sufficient time and effort has been spent on implementation, maintenance, and improvement of the system. However, we cannot rely on “time and effort” and just hope that positive outcomes of health IT implementation will happen after a while; in the meantime, negative outcomes may occur, which can endanger patients (e.g., hazards and medical errors), frustrate healthcare professionals, and impede their performance (e.g., burnout). We need to be vigilant and proactive in order to achieve benefits with health IT and avoid negative patient safety outcomes [5]. Human factors and human-centered design play a critical role in ensuring that health IT is well designed and fits with clinical and patient workflows.

We conducted a review of the literature on human factors and usability of health IT published between 2016 and 2019 in various databases: PubMed/Medline, Google Scholar, and PsycINFO. We also searched recent tables of content of major health informatics journals, such as JAMIA, IJMI, ACI, and Health Informatics. Building on chapters published in previous IMIA Yearbooks [6]–[8], we continue to see reports of usability problems with health IT. Emerging literature shows that development and implementation of human-centered design methods to use existing IT infrastructure can create value for patients and clinicians, for instance, in implementing usable technologies, e.g., clinical decision support (CDS) for supporting clinical decision making and care coordination. This chapter also highlights new issues in human-centered design of health IT, such as design of health IT for teams.

 

MissingLinkText

2 Usability Problems of Health IT and Their Negative Impact on Patients and Clinicians

An extensive account of patient safety issues with health IT was recently published in the US trade literature with a dramatic title: “Death By 1,000 Clicks: Where Electronic Health Records Went Wrong” [9]. The authors provided examples of poor health IT usability experienced by physicians and their impact on patient outcomes. Health IT usability problems experienced by physicians, nurses, and other health care professionals can produce multiple negative outcomes for both patients (e.g. threats to patient safety) and clinicians (e.g. frustration, work-arounds, burnout) [2], [10]–[12].

2.1 Impact of Health IT on Patient Safety

We continue to see evidence of the linkage between poor design of health IT and patient safety consequences. An analysis of medication safety events reported by pediatric clinicians in three children hospitals across the US showed that more than one-third (36%) were related to EHR usability issues, in particular the lack of system feedback and poor visual display of information [13]. The alarming statistic of this study is that 19% of the EHR usability-related events could have resulted in patient harm. A prospective study of medication safety among 624 patients in two ICUs of a medical center identified a total of1,622 preventable adverse drug events after EHR implementation [14]. Thirty-four percent of medication events were related to EHR technology; they included orders with missing information and accepted duplicate medication orders. These findings echo conclusions by Kushniruk et al. [6], [7] in previous IMIA Yearbooks that the lack of attention to human factors with health IT and the failure to engage end-users in the design and implementation processes produce negative consequences for patient safety. Users’ participation and analysis of their work are key elements of the human-centered design process. Increased recognition of the benefits of deep analyses of clinical workflow in this human-centered design process fits with the focus of resilience engineering on the actual work done as opposed to “work-as-imagined” [15], [16].

 

MissingLinkText

2.2 Impact of Health IT on Clinician Quality of Working Life

In addition to the impact of poor health IT usability on patient safety, there is increasing attention on the negative impact of health IT use on clinicians. Unmanageable demands for clinical documentation and poor health IT usability and workflow integration are some of the EHR-related factors that have been suggested as contributing to clinician burnout. In a survey of more than 5,000 physicians across all specialties in the US, Shanafelt et al. [11] found that the use of EHR and CPOE technologies by physicians was associated with lower physician satisfaction with time spent on clerical tasks and a higher likelihood of professional burnout. A study of EHR event log data showed that primary care physicians spent almost six hours per day interacting with the EHR both during and after clinic hours, therefore, indicating potential challenges with workload and interferences between the professional and private life spheres [17]. A survey of 4,197 physicians confirms the challenges and potential stressful impact of health IT [18]. Sixty-four percent of the physicians agreed or strongly agreed that the EHR added to their daily frustration, 38% reported that time spent on the EHR at home was moderately high or excessive, and 46% indicated that time spent on documentation was “poor or marginal”. All of these EHR-related stressors were significant predictors of physician burnout. Clinicians have the feeling that they put a lot of time and effort to support technology, whereas technology should support them in their clinical activities [1].

 

MissingLinkText

2.3 Old and New Challenges for Human Factors and Usability of Health IT

Studies on the impact of poor health IT design (and implementation) for both patients and clinicians are important; but we need to better understand the linkage between health IT usability and patient safety. We also need to move beyond analyzing health IT usability and its impact on patient safety, and create more evidence for actually improving the design of health IT. If we want human factors to be taken seriously into account, we should not be shouting from the sideline, but get actively involved in the design and implementation of health IT, and evaluate the impact of our human factors methods and principles on the technology in practice. This research evidence should include information on how to design better (more usable) technologies, as well as their positive impact on key outcomes, e.g., patient safety. Section 4 provides examples of human-centered design applications that produce benefits in usability as well as improvements in care process and patient outcomes. We end the chapter with a discussion of the emerging challenge of designing health IT for teams.

 

MissingLinkText

 

MissingLinkText

3 Improved Understanding of Health IT Usability and How It Is Linked to Patient Safety

Two recent viewpoints in JAMA raised the alarm about usability of health IT that “remains suboptimal” and provided policy recommendations to deepen and broaden the application of human factors methods and principles in health IT [19], [20]. Deeper understanding of how usability of health IT can actually be improved is at the core of these recommendations. This improvement will require that we more clearly identify where and when usability gets created or breaks down, how usability is linked to patient safety, and how usability methods are used (and not used) by health IT designers and implementers.

3.1 Improving Usability of Health IT at Design and Implementation Stages

Recent research is helping us to better understand the problem of health IT usability and its relation to patient safety. Usability of health IT is created (or influenced) at both stages of design (e.g., IT vendors) and implementation (i.e. software configuration by health care organizations). Early research showed the lack of systematic integration and use of human factors methods by health IT vendors [21], [22]. Recently, Ratwani et al. [23] documented variations in health IT usability at the level of health care organizations. Using screen capture software, they recorded EHR tasks performed by Emergency Department (ED) physicians in four different hospitals that used two different EHR technologies (i.e., Epic and Cerner). At each hospital, between 12 and 15 ED physicians participated in the study; each physician performed a total of six tasks: two diagnostic imaging orders, two laboratory orders, and two electronic prescriptions. The researchers extracted the following usability measures from the videos: total task duration, number of clicks, and error rate. Results showed large variation in all three usability measures across the two EHR technologies as well as across the hospitals. For instance, the error rate varied between 17% and 50% for the ordering of prednisone taper, between 0% and 30% for ordering Tylenol, and was consistently 0% for entering the lab order of lactate. These data confirm the role of implementation processes in influencing usability. Health care organizations make decisions about customization, configuration, and other implementation processes (e.g., training, user input) that can have significant influence on the final usability of the EHR technology in use. Therefore, improving usability and other human and organizational aspects of health IT will require action and effort by both designers and implemented of health IT.

 

MissingLinkText

3.2 Developing Better Understanding of Linkage Between Usability and Patient Safety

In addition to understanding that usability is constructed by various actors involved in the design and implementation of health IT, we are developing a deeper knowledge of the mechanisms by which usability influences patient safety. Research by the French human factors group at the University of Lille is producing important knowledge about the impact of usability on patient safety; this is critical research as it helps to untangle the “causal chain” between the technology and outcomes, including patient safety [24] [25] [26]. Watbled and colleagues [26] provided empirical evidence for the linkage between usability flaws (i.e., violations of usability principles), usage problems, and negative outcomes in the context of health IT for discharge summaries. The researchers used three usability evaluation methods: heuristic evaluation based on Scapin and Bastien’s method [27], user testing in an experimental setting, and in-situ observations of the technologies’ use after implementation. They identified multiple usability flaws in the two technologies, and documented the chain of evidence between usability flaws, usage problems, and negative outcomes. For instance, the heuristic evaluation identified the need for additional steps to insert a dictated discharge summary in the electronic medical record. This usability flaw led to a usage problem, as additional tasks were needed. Because of the increased workload, there was a potential negative outcome of inserting the wrong discharge summary in a given patient’s record. This type of in-depth analysis helps to identify the specific usability flaws that can have negative consequences on patient safety, therefore, helping to focus on high-risk design issues that need to be addressed by designers and/or implementers.

 

MissingLinkText

3.3 Need to Understand Work System of IT Designers and Implementers

As we are developing a deeper knowledge about health IT usability and its role in patient safety, we are clearly justifying the need for more and better application of human factors methods and principles in the design and implementation of health IT. However, a major open issue remains: our lack of knowledge about the work of health IT designers and implementers. This issue represents an interesting dilemma for the human factors community as a core principle of our discipline is to first understand the work situation and then to use this knowledge to develop tools and technologies to support the actual work; but we have not yet applied our human factors approaches to analyze the work of health IT designers. We know very little about the work of health IT designers and implementers; without knowledge on their work context, it remains difficult to elaborate recommendations on how to integrate human factors in the health IT development life cycle.

Outside of health care, research has been conducted to examine the actual work of designers and engineers and their experience of constraints, demands, and stress, which may limit their ability to apply human factors methods and principles systematically while designing, for instance, off-shore oil platforms [28], [29]. Ironically, Charles Perrow, one of the fathers of safety, and the author of “Normal Accidents” [30], conducted the first study on the organizational context of human factors engineering implementation [31], and identified several organizational factors (e.g., lack of support from top management) that can limit the impact of human factors engineers on the design of technologies and equipment. We need to do justice to our own human factors discipline and apply our human factors methods to develop knowledge about the work system of health IT designers and implementers; this knowledge is a critical open research area that will contribute to the effective translation of our human factors knowledge into practice [6]. This research would need to address multiple barriers, such as researchers’ access to health IT vendors and trust between designers, implementers, and researchers [20].

 

MissingLinkText

 

MissingLinkText

4 Human-centered Design of Health IT

In previous editions of the IMIA Yearbook, authors emphasized the importance of embedding human factors methods in the early stage of health IT [7], [32] and they described approaches for implementing human-centered design for health IT [32], [33]. We continue to see research that shows the positive impact of human-centered design on usability, as well as important care process and patient outcomes. This research is necessary to demonstrate the “value” of human factors-based approaches in health IT design.

4.1 Impact of Human Factors Engineering in Health IT Design on Usability

We continue to see studies that implement various human factors methods and principles to design health IT applications and demonstrate benefits on various usability dimensions. For instance, Savoy et al. [34], [35] developed a deep understanding of cognitive requirements related to primary care physicians or specialists requesting a consultation; this was accomplished using a combination of semi-structured interviews and direct observations [35]. An interdisciplinary team (including human factors engineers) developed design guidelines for each cognitive requirement, which were then used to develop a prototype. A within-subject experiment of the prototype in comparison to the usual EHR consultation template showed improvement in multiple usability dimensions: perceived satisfaction, fewer mouse clicks, and faster completion but only for one of the subtasks in the consultation request process [34]. A team of ED clinicians and human factors engineers applied cognitive systems engineering methods to develop an ED information system prototype [36]. A usability evaluation of the prototype with 18 ED clinicians (attending physicians, resident physicians, physician assistants, and nurses) provided strong evidence that the ED information system prototype supported a range of cognitive activities, e.g., patient identification with specific characteristics (e.g., longest waiting time in the ED) and assessment of resources for current patient demands. The study participants rated the prototype’s usability and usefulness very high.

Studies are also published on the role of human factors engineering in designing usable health IT for patients, such as a diabetes dashboard embedded in a patient portal [37], an app for geriatric heart failure self-care [38], and an app for medication management [39]. The domain of consumer health IT has embraced human factors and usability methods [40]. Participation of patients and caregivers in health IT design processes can not only help to produce more usable technologies, but also lead to other benefits and patient-centered outcomes such as empowerment and shared decision making [41].

Studies demonstrate that critical applications of human factors engineering in the early stage of health IT design lead to benefits for both clinicians and patients. However, the question remains about dissemination of human factors knowledge on a larger scale. Brunner et al. [42] provided a unique perspective on that question when they examined implementation of user-centered design practices in 170 United States Veteran Affairs primary care clinics. They assessed whether primary care clinics were using the following practices when developing computerized clinical reminders and/ or disease-specific templates, i.e., CDS: (1) pilot testing of CDS before full-scale implementation, (2) assessment of provider satisfaction post-implementation, (3) formal evaluation of CDS usability, and (4) analysis of the impact of CDS on performance. The implementation of these four practices was uneven across the 170 primary care clinics: 74% implemented pilot testing, 62% carried out the assessment of post-implementation provider satisfaction, 36% performed formal usability assessment, and 79% did the analysis of the CDS impact. The practice “analysis of CDS impact on performance improvement” was associated with a greater utility of the CDS as perceived by primary care clinic directors. This research helps us to understand the organizational context (e.g., primary care clinic) in which usability and other human factors methods can (should) be implemented.

 

MissingLinkText

4.2 Value of Human Factors Engineering in Health IT Design in Improving Care Process and Patient Outcomes

A few studies are emerging to show the actual value of incorporating human factors methods and principles in the health IT design process, demonstrating impact on key care process and patient outcomes. This is a critical step forward in demonstrating the value of human factors engineering as we show that human factors approaches “make a difference” in improving care processes and patient outcomes, such as medication safety [43] and appropriate medication selection during a code [44]. An interdisciplinary team at Johns Hopkins University in the US (including a human factors engineer) applied two rounds of usability testing to significantly reduce the incidence of duplicate medical records from 4% to 2.3% [45]. The improvement was sustained even two years after the new medical record creation process was implemented. Round 1 of usability testing included the mapping of the patient search and selection process used by registrars, the direct observation of registrars to understand their search behaviors, and the identification of barriers experienced by registrars in searching patient names. Results of this round led to multiple modifications to the patient name search process, which were tested in the second round of usability testing. This study confirms the need for an iterative process with multiple methods and approaches in order to ensure good usability of health IT [7]. A critical aspect of this study is demonstrating the impact of using human factors methods on key care processes and outcome measures, such as reduction of the number of duplicate medical records. We need more research that demonstrates the value of human factors engineering for the design of health IT in improving care process and patient outcomes.

 

MissingLinkText

4.3 Design of Health IT for Teams

As team-based care is increasingly implemented in the continuum of patient care [46], we are seeing an unfortunate mismatch between health IT designed for individual tasks and what we need, i.e., health IT designed for care processes and teams [47]. To design technologies for teams, we need a better understanding of the critical information that team members need to share and communicate for shared team awareness [48], [49]. A couple of research projects focus on the application of human factors engineering to design health IT for teams. Two research teams in Ottawa, Canada [48], [50]–[52] and at the University of Illinois [53] used various human factors methods that provided input to build shared team displays for supporting cognitive work and communication during resuscitations. In Canada, Parush et al., designed a smart display to support cognitive work and team communication during ED resuscitation [48], [50]–[52]. Using multiple methods (e.g., interview of ED clinicians, observation of live resuscitations), they identified key information needs, and then implemented an iterative human-centered design process to develop a prototype for the shared situation display [48], [52]. They tested the display in a simulator environment with three teams made of attending physicians, residents, nurses, and respiratory therapists. Resuscitation teamwork was rated using the Clinical Team Scale (CTS). CTS scores were higher when teams used the shared display than when they did not use it [54]. When teams did not use the shared display, they spent more time exchanging information on the patient status; when they used the display, they spent more time discussing the treatment for the patient. This provided some evidence that the shared status display helped the team in focusing on the patient’s treatment rather than on just exchanging information. A qualitative follow-up analysis of data gathered in the experiment debriefs and focus groups studies confirmed how the shared display helped to build team situation awareness during resuscitations [50].

Wu et al. [53] used the contextual design methodology [55] to develop initial design requirements for a shared display to support cognitive work (i.e., information assembling, recall, and real-time tracking) in cardiac arrest resuscitations. They conducted three simulation sessions with ICU physicians and nurses who were also interviewed post-simulation. This iterative process helped to update the list of design requirements and provided feedback on the design of the shared display. The final stage of this human-centered design process consisted of three scenario-based simulations with 12 ICU physicians and nurses using a mannequin in a simulator. Different physicians and nurses participated in the three simulations in order to increase the diversity of perspectives and inputs in the design process. When using the shared display, physicians and nurses reported lowers levels of mental workload. During the debriefing interviews, they indicated the reasons why and how the shared display could improve their performance. For instance, they mentioned that the system could improve their ability to adhere to best practice guidelines for resuscitation, as well as their common (and shared) understanding of the treatment progression.

There is an urgent need to develop research and knowledge about how to design health IT that can support the work of teams, in particular care teams that are distributed over time and space along the patient journey [56]–[58]. This research requires a shift in focus from the individual to the team, as well as innovative methods to evaluate the impact of these team-based technologies on clinical care processes and team functioning. Several studies [43], [52], [53] have shown how scenario-based simulation can be used to evaluate the impact of human factors-based design of health IT on important clinical outcomes (e.g., prescription errors). We should extend the evaluation of health IT design beyond individual “subjective” measures such as end-user satisfaction, and include team outcomes such as shared understanding and team communication, as well as “objective” measures, e.g., task duration, number of clicks and scrolls needed to navigate through the technology, and proxy clinical outcomes such as the number of medication errors, the number of correct diagnoses, and appropriate clinical decision making. We need further research efforts to develop innovative human factors and usability methodologies such as those relying on multiple methods, including sensors and EHR-based methods [59].

 

MissingLinkText

 

MissingLinkText

5 Conclusion

Recent literature on human factors and organizational aspects of health IT continues to bring up problems of usability, poor workflow integration, and potential negative impact on patient safety and other patient and clinician outcomes (e.g., frustration, burnout). It is encouraging to see emerging and growing literature on the actual role of human factors engineering in designing more usable and safer health information technologies. This research needs to grow and address critical aspects of the changing landscape of health care, including designing health IT for teams. Designing usable health IT for individual tasks of clinicians or patients is challenging, but designing health IT for teams with various needs, perspectives, and motivations is even more challenging. Further understanding how the collaborative health IT design process can integrate multiple perspectives is paramount [60].

Recent media attention to the problem of health IT usability and its negative impact on patient safety and clinicians [1], [9] may provide the impetus for increased resources dedicated to health IT usability and the broad application of human factors methods and principles in the design and implementation of health IT. In order to respond to the call for increased integration of human factors in health IT design, we need to better understand the actual work performed by health IT designers. This can be done by applying human factors method and principles to the work of health IT design and implementation: applying human factors approaches to human factors in practice (i.e., health IT design and implementation) is needed.

 

MissingLinkText

 

MissingLinkText

Acknowledgments

This research was supported by the Clinical and Translational Science Award (CTSA) program, through the NIH National Center for Advancing Translational Sciences (NCATS), Grant UL1TR000427. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Funding for this research was also provided by the Agency for Healthcare Research and Quality (AHRQ) [Grant No. R01HS022086].

• References

• 1 Gawande A. Why doctors hate their computers. The New Yorker; 2018
• 2 Staggers N, Elias BL, Makar E, Alexander GL. The imperative of solving nurses’ usability problems with health information technology. J Nurs Adm 2018; 48 (04) 191-6
• 3 Czaja SJ, Zarcadoolas C, Vaughon WL, Lee CC, Rockoff ML, Levy J. The usability of electronic personal health record systems for an underserved adult population. Hum Factors 2015; 57 (03) 491-506
• 4 Lin SC, Jha AK, Adler-Milstein J. Electronic health records associated with lower hospital mortality after systems have time to mature. Health Aff (Millwood) 2018; 37 (07) 1128-35
• 5 Agboola SO, Bates DW, Kvedar JC. Digital health and patient safety. JAMA 2016; 315 (16) 1697-8
• 6 Turner P, Kushniruk A, Nohr C. Are we there yet? Human factors knowledge and health information technology - the challenges of implementation and impact. Yearb Med Inform 2017; 26 (01) 84-91
• 7 Kushniruk A, Nohr C, Borycki E. Human factors for more usable and safer health information technology: Where are we now and where do we go from here?. Yearb Med Inform 2016 (01) 120-5
• 8 Patel VL, Kannampallil TG. Human factors and health information technology: Current challenges and future directions. Yearb Med Inform 2014; 9: 58-66
• 9 Schulte F, Fry E. Death by 1,000 clicks: Where electronic health records went wrong. Kaiser Health News; March 18, 2019
• 10 Kutney-Lee A, Sloane DM, Bowles KH, Burns LR, Aiken L. Electronic health record adoption and nurse reports of usability and quality of care: The role of work environment. App Clin Inform 2019; 10 (01) 129-39
• 11 Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J. , et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin Proc 2016; 91 (07) 836-48
• 12 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med 2013; 173 (08) 702-4
• 13 Ratwani RM, Savage E, Will A, Fong A, Karavite D, Muthu N. , et al. Identifying electronic health record usability and safety challenges in pediatric settings. Health Aff (Millwood) 2018; 37 (11) 1752-9
• 14 Carayon P, Du S, Brown R, Cartmill R, Johnson M, Wetterneck TB. EHR-related medication errors in two ICUs. J Healthc Risk Manag 2017; 36 (03) 6-15
• 15 Braithwaite J, Wears RL, Hollnagel E. Resilient health care: Turning patient safety on its head. Int J Qual Health Care 2015; 27 (05) 418-20
• 16 Hollnagel E. Human factors/ergonomics as a systems discipline? “The human use of human beings” revisited. Appl Ergon 2014
• 17 Arndt BG, Beasley JW, Watkinson MD, Temte JL, Tuan WJ, Sinsky CA. , et al. Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med 2017; 15 (05) 419-26
• 18 Gardner RL, Cooper E, Haskell J, Harris DA, Poplau S, Kroth PJ. , et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Asoc 2019; 26 (02) 106-14
• 19 Ratwani RM, Reider J, Singh H. A decade of health information technology usability challenges and the path forward. JAMA 2019
• 20 Ratwani RM, Hodgkins M, Bates DW. Improving electronic health record usability and safety requires transparency. JAMA 2018
• 21 McDonnell C, Werner K, Wendel K. Electronic Health Record Usability: Vendor Practices and Perspectives. Rockville, MD: Agency for Healthcare Research and Quality; 2010. Contract No.: AHRQ Publication No. 09(10)-0091-3-EF
• 22 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: Analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-82
• 23 Ratwani RM, Savage E, Will A, Arnold R, Khairat S, Miller K. , et al. A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inform Assoc 2018; 25 (09) 1197-201
• 24 Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-Zephir MC. Usability flaws in medication alerting systems: Impact on usage and work system. earb Med Inform 2015; 10 (01) 55-67
• 25 Marcilly R, Ammenwerth E, Vasseur F, Roehrer E, Beuscart-Zephir MC. Usability flaws of medication-related alerting functions: A systematic qualitative review. J Biomed Inform 2015; 55: 260-71
• 26 Watbled L, Marcilly R, Guerlinger S, Bastien JMC, Beuscart-Zephir M, Beuscart R. Combining usability evaluations to highlight the chain that leads from usability flaws to usage problems and then negative outcomes. J Biomed Inform 2018; 78: 12-23
• 27 Scapin DL, Bastien JMC. Ergonomic criteria for evaluating the ergonomic quality of interactive systems. Behaviour & Information Technology 1997; 16 (04) 220-31
• 28 Wulff IA, Westgaard RH, Rasmussen B. Ergonomic criteria in large-scale engineering design - I - Management by documentation only? Formal organization vs. designers’ perceptions. Appl Ergon 1999; 30: 191-205
• 29 Wulff IA, Westgaard RH, Rasmussen B. Ergonomic criteria in large-scale engineering design - II - Evaluating and applying requirements in the real-world of design. Appl Ergon 1999; 30: 207-221
• 30 Perrow C. Normal Accidents - Living with HighRisk Technologies. New York: Basic Books; 1984
• 31 Perrow C. The organizational context of human factors engineering. Adm Sci Q 1983; 28 (04) 521
• 32 Beuscart-Zephir MC, Elkin P, Pelayo S, Beuscart R. The human factors engineering approach to biomedical informatics projects: State of the art, results, benefits and challenges. Yearb Med Inform 2007; 46 (Suppl 1): 109-27
• 33 Beuscart-Zephir MC, Borycki E, Carayon P, Jaspers MWM, Pelayo S. Evolution of human factors research and studies of health information technologies: the role of patient safety. Yearb Med Inform 2013; 8 (01) 67-77
• 34 Savoy A, Patel H, Flanagan ME, Daggy JK, Russ AL, Weiner M. Comparative usability evaluation of consultation order templates in a simulated primary care environment. Appl Ergon 2018; 73: 22-32
• 35 Savoy A, Militello LG, Patel H, Flanagan ME, Russ AL, Daggy JK. , et al. A cognitive systems engineering design approach to improve the usability of electronic order forms for medical consultation. J Biomed Inform 2018; 85: 138-48
• 36 Guarrera TK, McGeorge NM, Clark LN, LaVergne DT, Hettinger ZA, Fairbanks RJ. , et al. Cognitive engineering design of an emergency department information system. In: Bisantz AM, Burns CM, Fairbanks RJ. editors. Cognitive Systems Engineering in Health Care. CRC Press; 2014: 43-74
• 37 Martinez W, Threatt AL, Rosenbloom ST, Wallston KA, Hickson GB, Elasy TA. A patient-facing diabetes dashboard embedded in a patient web portal: Design sprint and usability testing. JMIR Hum Factors 2018; 5 (03) e26
• 38 Srinivas P, Cornet V, Holden R. Human factors analysis, design, and evaluation of Engage, a consumer health IT application for geriatric heart failure self-care. Int J Hum Comput Interact 2017; 33 (04) 298-312
• 39 Sedlmayr B, Schoffler J, Prokosch HU, Sedlmayr M. User-centered design of a mobile medication management. Inform Health Soc Care 2018; 1-12
• 40 Valdez RS, Holden RJ, Novak LL, Veinot TC. Transforming consumer health informatics through a patient work framework: Connecting patients to context. J Am Med Inform Assoc 2015; 22 (01) 2-10
• 41 Griffioen I, Melles M, Stiggelbout A, Snelders D. The potential of service design for improving the implementation of shared decision-making. Design for Health 2017; 1 (02) 194-209
• 42 Brunner J, Chuang E, Goldzweig C, Cain CL, Sugar C, Yano EM. User-centered design to improve clinical decision support in primary care. Int J Med Inf 2017; 104: 56-64
• 43 Russ AL, Zillich AJ, Melton BL, Russell SA, Chen S, Spina JR. , et al. Applying human factors principles to alert design increases efficiency and reduces prescribing errors in a scenario-based simulation. J Am Med Inform Assoc 2014; 21 (e2): e287-e96
• 44 Rousek JB, Hallbeck MS. Improving medication management through the redesign of the hospital code cart medication drawer. Hum Factors 2011; 53 (06) 626-36
• 45 Khunlertkit A, Dorissaint L, Chen A, Paine L, Pronovost PJ. Reducing and sustaining duplicate medical record creation by usability testing and system redesign. J Patient Saf. In press
• 46 Mitchell P, Wynia M, Golden R, McNellis B, Okun S, Webb EC. , et al. Core Principles & Values of Effective Team-Based Health Care. Washington, DC: Institute of Medicine; 2012
• 47 Walker J, Carayon P. From tasks to processes: The case for changing health information technology to improve health care. Health Aff (Millwood) 2009; 28 (02) 467
• 48 Parush A. Displays for health care teams: A conceptual framework and design methodology. In: Bisantz AM, Burns CM, Fairbanks RJ. editors. Cognitive Systems Engineering in Health Care. CRC Press; 2014: 75-96
• 49 Wooldridge AR, Carayon P, Hoonakker P, Hose BT, Ross J, Kohler JE. , et al. Complexity of the pediatric trauma care process: Implications for multi-level awareness. Cogn Technol Work In press
• 50 Calder LA, Bhandari A, Mastoras G, Day K, Momtahan K, Falconer M. , et al. Healthcare providers’ perceptions of a situational awareness display for emergency department resuscitation: A simulation qualitative study. Int J Qual Health Care 2018; 30 (01) 16-22
• 51 Parush A, Kramer C, Foster-Hunt T, McMullan A, Momtahan K. Exploring similarities and differences in teamwork across diverse healthcare contexts using communication analysis. Cognition, Technology & Work 2014; 16 (01) 47-57
• 52 Parush A, Mastoras G, Bhandari A, Momtahan K, Day K, Weitzman B. , et al. Can teamwork and situational awareness (SA) in ED resuscitations be improved with a technological cognitive aid? Design and a pilot study of a team situation display. J Biomed Inform 2017; 76: 154-61
• 53 Wu P, Nam MY, Choi J, Kirlik A, Sha L, Berlin Jr RB. Supporting emergency medical care teams with an integrated status display providing real-time access to medical best practices, workflow tracking, and patient data. J Med Syst 2017; 41 (12) 186
• 54 Guise JM, Deering SH, Kanki BG, Osterweil P, Li H, Mori M. , et al. Validation of a tool to measure and promote clinical teamwork. Simul Healthc 2008; 3 (04) 217-23
• 55 Beyer H, Holtzblatt K. Contextual Design: Defining Customer-Centered Systems. San Francisco, CA: Morgan Kaufmann Publishers; 1997
• 56 Carayon P, Wooldridge A. Improving patient safety in the patient journey: Contributions from human factors engineering. In: Smith AE. editor. Women in Industrial and Systems Engineering: Key Advances and Perspectives on Emerging Topics. Springer Nature; 2019
• 57 Carayon P, Wood KE. Patient safety: The role of human factors and systems engineering. In: Rouse WB, Cortese DA. editors. Engineering the System of Healthcare Delivery. IOS Press; 2009: 23-46
• 58 Hoonakker PLT, Wooldridge AR, Carayon P, Hose BZ, Eithun BL, Brazelton TB. et al., editors. Things Falling Through the Cracks: Information Loss During Pediatric Trauma Care Transitions. Proceedings of the 20th Congress of the International Ergonomics Association (IEA 2018), Florence, Italy. Cham, Switzerland: Springer; 2018
• 59 Calvitti A, Hochheiser H, Ashfaq S, Bell K, Chen Y, El Kareh R. , et al. Physician activity during outpatient visits and subjective workload. J Biomed Inform 2017; 69: 135-49
• 60 Détienne F. Collaborative design: Managing task interdependencies and multiple perspectives. Interact Comput 2006; 18 (01) 1-20