Avoiding Unintended Consequences of Pediatric Blood Order Set Updates through In Situ Usability Testing

Abstract

Background Blood product ordering is a complex process and mistakes can harm patients and lead to poor outcomes. Orders and order sets can be designed to help mitigate errors, but major changes in design can unintentionally cause new errors.

Objectives Our objective was as follows: (1) utilize formative in situ usability testing to iteratively improve the design of a redesigned blood product order set prior to go-live, (2) implement changes based on feedback derived from this testing, and (3) compare the error rate, system usability scale (SUS) score, time to task completion, and click counts between the prior order set in use at the time and the revised redesigned order set.

Methods A multidisciplinary project team convened to redesign blood product orders and order sets from scratch based on a review of the literature and benchmarking against four pediatric academic institutions with the goal of addressing prior ordering errors. The newly redesigned blood product order set was iteratively updated via in situ formative usability testing performed with available clinical users using a concurrent think-aloud protocol in real clinical environments. Errors, SUS scores, time to task completion, and click counts were assessed for the revised redesigned order set using summative testing.

Results Formative usability testing with 20 participants led to seven design changes in the redesigned order set which reduced the error rate at go-live. Summative usability testing showed that even though the usability scores were only slightly improved for the revised redesigned order set, the error rates in blood orders were significantly decreased.

Conclusion Usability testing can identify design errors early in the process which can be rectified prior to implementation, thus avoiding unintended consequences of changes.

Keywords

user-centered design - special transfusion - preventable safety - usability testing - VSA

Background and Significance

Overtransfusion of blood products and transfusion of products with incorrect product modifications are serious but preventable safety hazards.[1] [2] Pediatric blood product ordering is more complex and thus prone to errors as compared with adults due to the variability in weight of the patient and the special transfusion considerations in neonates. Children are therefore at higher risk of adverse outcomes because they may not tolerate errors that would be less impactful to adults.[3] Blood product orders and order sets have been the subject of multiple studies within adult and pediatric organizations due to their complexity combined with the risk to the patient if an error in ordering occurs. However, while user-centered design with usability testing has been studied previously with blood product ordering in this institution and in one other institution for massive transfusion protocols to ensure design quality and function,[4] [5] [6] [7] [8] other publications regarding this practice were not found in the medical literature. This previous work consolidated different order sets into one general order set for the vast majority of blood product orders (herein referred to as the “primary order set”), added clinical indications next to requests for product modifications to assist providers with selecting the correct modifications, supported aliquot ordering, and simplified the overall structure of the order set. However, despite the successes of previous work, the system continued to experience several errors related to overtransfusion, incorrect product modifications, incorrect emergency blood orders, and confusion regarding aliquot orders between the ordering clinicians and the blood bank technologists and physicians. In response, a large multidisciplinary value stream analysis (VSA) committee was convened to improve the overall blood product ordering and administration process which included further simplification of blood orders and order sets. High institutional priority to fix the problem quickly was balanced against the risk of large changes to a complex sociotechnical system.[9] [10] We combined Lean and Function Resonance Analysis Method (FRAM) to understand the blood product ordering and administration, the results of which were presented previously.[11] Based on this work, the VSA committee determined that a redesign of all blood product orders and many of the blood product order sets from scratch was necessary to improve their understandability and efficiency and thereby reduce errors. While the details of the redesigned orders and order sets as well as process changes will be submitted separately for publication, the impact of formative and summative usability testing on this redesign has not been otherwise covered and was a critical component to the success of this initiative because changes were made to the initial redesign based on the results of usability testing which significantly contributed to the success of the implementation. The objectives of using usability testing in this study were to (1) evaluate the initial redesign of the orders and primary order set to clinical end-users via in situ formative usability testing, (2) make revisions to the initial redesigns based on feedback from formative usability testing with approval from the VSA steering committee, and (3) evaluate the effectiveness of the usability testing intervention via in situ summative usability testing on the revised redesigned orders and primary order set.

Methods

The VSA project was a significant institutional effort which involved multiple teams, each covering a different aspect of the blood product ordering process. These teams included physicians, advanced practice professionals (APPs), nurses, blood bank technologists and information system specialists from clinical informatics, transfusion medicine (pathology), laboratory, anesthesia, nursing, and hematology and oncology.

Selection of Changes to be Made to Orders and Order Sets

To develop the initial redesign, the Transfusion Committee team first brainstormed a list of items felt to be confusing by the clinical teams. This was compared against a review of the literature on blood product ordering. The Transfusion Committee team then solicited information from four external tertiary pediatric health care organizations for their blood product order parameters and processes.

During this phase, several important issues were targeted for improvement: (1) selection of product modifications for blood products, (2) selection and clarification of the correct product volume to prepare and/or transfuse, and (3) ensuring that transfusionists had adequate product releases available in the electronic health record (EHR) from which to launch positive patient identification for each blood product to be transfused.

Changes to Product Modifications

The prior blood product orders required physicians and APPs to select specific product modifications appropriate for the patient's condition(s) such as irradiation, phenotypically matched, cytomegalovirus negative, washed cells, etc. Even though ordering providers were asked about whether the patient had any particular clinical conditions associated with the relevant selections for product modifications, they reported uncertainty about what product modifications were needed. The providers frequently used historical product modification data to inform their current orders. However, the blood bank had a separate laboratory record with historical patient product modification data which ordering providers could not access. This resulted in confusion and additional calls for clarification between the blood bank and ordering providers. Errors in orders required the ordering clinician to cancel existing orders and place new orders with the correct information.

Pediatric organizations either ask for product modifications within the order or the modification indications. The trade-off between the two options is either additional work or decision-making for either the ordering provider or the blood bank. Acknowledging that knowing what product modifications to order by patient condition often requires advanced transfusion medicine knowledge, the Transfusion Committee agreed to fundamentally redesign blood product orders to remove product modification requests entirely. To this end, specific product attribute requests (e.g., irradiation and washing) were replaced with a multiselect question for clinical indications (e.g., suspected T cell deficiency and confirmed immunoglobulin A deficiency) that drove already existing protocol-based selection of product modifications by the blood bank.

Changes to Dosing Product Volumes

Organizations that have capacity to support just-in-time blood product requests have one to one prepare and transfuse orders which means the volumes match every time between orders. However, there are institutions where prepare and transfuse orders are not linked due to logistical reasons, like in this study. In such cases, this can lead to volume discrepancies between the prepare and transfuse orders. Further, the presence of aliquots in both the prior prepare orders and transfuse orders caused confusion between ordering clinicians, nurses, and blood bank technologists because it was not clear whether the total volume on the product dose was for each aliquot or for the combination of both aliquots. In prior work, alerts were added to transfusion orders when the volume to be transfused in nonemergency situations exceeded the recommended highest volume for the patient's dosing weight. In this redesign, additional strategies to prevent accidental overtransfusions included the elimination of aliquots, clarification of the dosing mechanism within the order, and interruptive alerts for high-volume prepare orders in addition to transfuse orders.

Changes to Transfusion Order Releases to Support Positive Patient Identification

The delinked prepare and transfuse order design led to challenges with insufficient releases available in the EHR to launch transfusion-positive patient identification. This was primarily due to normal and unpredictable variations in exact volumes between different units of the same product delivered from the blood supplier (e.g., one unit of red cells could be anywhere from 250–350 mL) contrasted with the need to precisely dose blood product in pediatric patients, including older patients who were volume sensitive. Consequently, additional changes were made to transfuse orders to ensure adequate transfuse orders (releases) for launching transfusion-positive patient identification for each product.

Additional Changes to Orders and Order Set Design

A sidebar report was added to the orders to provide relevant laboratory test results and transfusion history at the time of order. Nearly all individual blood product orders were updated, and 16 order sets required significant updates. This was a significant institutional effort that involved many people which were fully sanctioned by institutional leadership because of the need to address the errors and to help reduce physician and APP cognitive load.

Formative Usability Testing

Formative in situ usability testing was performed to evaluate the redesigned order set. Testing efforts focused on the primary order set as this is the most frequently used order set for the vast majority of blood product orders. The redesigned primary order set was created in a demonstration environment of the EHR that mirrors the live environment without any protected health information. Patients were created with specific ages and weights to reflect the created scenarios and to mimic what physicians and APPs would see in the live EHR environment. Four patient scenarios were created to test blood product ordering with an emphasis on this major design change using a demonstration EHR environment in a live clinical setting, herein referred to as in situ usability testing ([Table 1]). This type of testing was chosen for several reasons. First, it allows for the discovery of potential design problems that can occur with natural workflows in a clinical environment with frequent interruptions. Second, by allowing physicians and APPs to self-direct through common patient scenarios, areas needing improvement or clarification are more easily identified. Third, the varied perspectives of different end-users that can impact correct utilization in the environment where the orders are expected to be used can be collected. Fourth, it increases the number of testing participants compared with traditional office-based usability testing which requires more effort for the participants to attend.

Table 1 Patient simulation scenarios for blood ordering

Scenario	Purpose
A 1-mo-old patient who needed a blood transfusion due to low hemoglobulin/hematocrit.	Provider ability to place a “standard” blood product order.
A 4 y, 19-kg patient that needed a scenario directed 15 mL/kg of blood product.	Provider recognition that the nurse would need two “transfuse” orders as the volume is greater than 250 mL.
A 5 y, 20-kg patient who needs a scenario directed 5 mL/kg of blood.	Provider ability to order smaller volumes of blood when “prepare” orders are limited to unit ordering only. The “transfuse” orders allow for specific volume amounts.
A 3 y in need of an emergency transfusion.	Ability of physicians and APPs to recognize emergency blood release is non-EHR process (uses paper form).

Abbreviations: APPs, advanced practice professionals; EHR, electronic health record.

With the help of an expert institutional EHR trainer, a wide range of inpatient physicians and APPs were recruited with different levels of transfusion experience to participate, while they were at the point of clinical care and on clinical service (in situ). One investigator presented the use cases to the participants and conducted the study. A second investigator observed participants' interaction with the EHR and took notes. A concurrent think-aloud protocol was used, and the team observed physicians and APPs as they completed scenarios. At the end of each session, notes were member-checked with participants, and feedback was obtained on design.[12] As this usability testing was done as part of an operational project, recording of participants was not done limiting rigorous qualitative analysis of the feedback obtained during sessions. However, this allowed for immediate verification of the feedback notes informing iterative design changes.

Incorporation of Feedback from Formative Usability Testing

Feedback was iteratively incorporated back into the redesigned primary order set to improve usability. As the study was done in situ, there were no specific time limits on sessions. The session durations varied from 10 to 30 minutes based on participant availability. Participants were encouraged to provide feedback and sessions ended when participants no longer had feedback or had to prioritize clinical work. While participants availability restricted the number of scenarios tested during certain sessions, there were no testing scenarios without feedback or completion of member-checking. For pragmatic reasons, we did not do a comprehensive qualitative analysis, but the main ideas or pain points were validated with participants. At the end of each session, the two team members discussed which feedback required simple design adjustments, such as name changes to a button, and made changes between sessions. If the feedback required either more complex build or process change to the blood product ordering, such as no default to the transfuse in mL order, it was reviewed with the committee prior to making updates. We continued iterations until data saturation, that is, no new feedback or technical limitations to changes requested.

Summative Usability Testing

After the redesign was penultimately finalized, summative usability testing was performed to evaluate performance in simulation. For this phase, participants were similarly recruited in active clinical areas who could spare 10 minutes. Four standardized clinical use, case scenarios were provided in which blood orders had to be placed. The scenarios and the sequence in which physicians and APPs were presented with the prior state versus the revised redesigned state blood order set were randomized. Physicians and APPs were timed while placing orders, and the team reviewed any errors made during the ordering process. After completing the scenarios, physicians and APPs filled out a system usability scale (SUS)[13] to rate the prior versus revised redesigned state order sets. Qualitative feedback was also obtained. Error rates were compared using chi-square (Χ ²) tests, and SUS scores and time to task completion were compared using t-tests.

Results

Formative Usability Testing

There were a total of 20 participants (12 APPs, 5 attending physicians, 2 residents, and 1 fellow). There were 152 prepare and transfuse orders placed in simulation with 39 errors detected ([Table 2]), each of which fell into one of the following categories: (1) missing transfusion orders when transfusion was intended, (2) wrong transfusion rate, (3) wrong volume, (4) wrong clinical indication for product modification, (5) wrong product (e.g., platelets instead of packed red blood cells [PRBCs]), (6) wrong modality (e.g., placing emergency orders for nonemergencies), (7) wrong workflow (e.g., placing an order in the EHR when scenario required a paper requisition for emergency release), (8) no indication for maximum volume to transfuse, and (9) inappropriate selection of the number of transfuse releases for volumes that would be more than one unit. When possible, design modifications to the redesigned order set were made in the demonstration environment of the EHR by clinical informaticists between participants during formative testing to evaluate possible fixes for each error.

Table 2 Formative usability testing order errors by scenario

Scenario	Prepare order errors	Transfuse order errors	Total
Scenario 1	10% (2/19)	10% (2/19)	10% (4/39)
Scenario 2	5% (1/19)	42% (8/19)	24% (9/39)
Scenario 3	10% (2/19)	10% (2/19)	10% (4/39)
Scenario 4	58% (11/19)	58% (11/19)	58% (22/39)
Total	21% (16/76)	30% (23/76)	26% (39/152)

Design Modifications

The design modifications made in the demonstration environment of the EHR between participants during formative testing included (1) providing dosing volume options of 5, 10, or 15 mL/kg within the transfuse order to calculate the final volume to be transfused to match what already existed in the prepare order, (2) updating language in the transfuse order to clarify requirement and reason for number of releases, (3) removing defaults for releases in the transfuse order to enable active decision-making, (4) adding an order group for PRBCs in mL in children ≥ 20 kg for specific volumes and redirecting users to place specific volumes in the transfuse in units for patients ≥ 20 kg, (5) adding the volume calculation row for maximum volume for transfusion in all transfuse orders in units, (6) updating wording so that all blood products start with different letter (e.g., removing “packed” from PRBCs to avoid confusion with platelets), and (7) including an option for emergencies that redirected physicians and APPs to use paper the required paper requisition ([Table 3]). No modifications were made to address missing transfusion orders as an alert already existed to remind providers to place a transfuse order when signing a prepare order without a corresponding transfuse order. No additional modifications were made to the transfusion rate as the current rate options were clear to providers; the issue was related to provider preference for a faster transfusion rate. The ability to make changes in real-time helped to reduce time spent on repeated errors and helped identify other potential errors more efficiently.

Table 3 Issues identified and remediated during formative usability testing

Candidate design from committee	Issues identified during usability	Design changes
• Single row with total volume free text.	• Pediatric volume calculation is mainly done in mL/kg; users felt that the transfuse order volume should mirror the prepare order volume in allowing calculation as mL/kg as opposed to total volume. • Users were frustrated with having to reenter blood product volume for mL orders.	• Provided volume options of 5, 10, or 15 mL/kg within the transfuse order to match the design of the prepare order • More blood is prepared than transfused and the orders are not connected making it technically impossible to autofill the transfuse order with the prepare order amount
• Language for transfuse options:  – “Volume > 250 mL”  – “Volume 250–500 mL”  – “Volume 500–750 mL”	• Users did not understand that the volume correlated to the number of transfuse orders or “releases” available to nurses.	• Update language to “1 release (vol < 250 mL),” “up to 2 release (vol 250–500 mL)” etc.
• Transfuse frequency defaulted to “volume > 250 mL”	• Users did not look at default release volume when adjusting volumes.	• Removed default for releases to enable active decision-making
• Order groups for “Non-OR (operating room) blood products” for patients ≥ 20 kg  – Red blood cells in units  – Platelets in units  – Frozen plasma (FFP) in units  – Cyroprecipitate in units  – Granulocytes (pt dosing weight ≥ 20 kg)	• Users were getting stuck in attempting to order a small volume (5 mL/kg) for patients ≥ 20 kg	• Added in an “order group” for PRBCs in mL in children ≥20 kg which redirects users to place orders in units and indicates volume in the transfuse order
• For “transfuse in unit” orders added option for maximum volume to transfuse • Original language stated “MAXIMUM mL/kg to infuse regardless of size of unit”	• User expressed concern for overtransfusion due to variability in unit sizes and the ability to no longer order in mL. • Users were concerned regarding the volume showing to nursing • Some confusion around wording	• Maximum volume options autocalculate volume for users • Updated language for volume amount to state “DO NOT EXCEED this volume in mL”
• Order groups for “non-OR blood products” for patients < 20 kg  – Packed red blood cells in mL  – Platelets in mL  – FFP in mL  – Cyroprecipitate in units  – Granulocytes < 20 kg	• User reported that she often accidently ordered platelets instead of PRBCs due to the “P”	• Updated wording so that all blood products start with a different letter.
• Informational red text at the top of the order set instruction users to use paper downtime form for emergency blood order	• Users did not perceive that emergency ordering should take place out of EHR	• Created a separate order group with a picture of the emergency release document and clear instructions to use the form

Abbreviations: EHR, electronic health record; PRBC, packed red blood cells.

Summative Usability Testing

A total of 21 physicians and APPs (5 APPs, 5 attending physicians, 5 residents, and 5 fellows) participated in summative testing performed on the redesigned order set which included the fixes described above. The error rate decreased significantly from 26% (11/42) to 12% (4/42) when using the revised redesigned order set (p = 0.04). Usability improved slightly from 78 (Grade B) to 84 (Grade A) but was not significantly different (p = 0.15). The time-to-task completion between the prior and revised redesigned state order sets was stable (pre: 113 seconds, post: 120 seconds, p = 0.42; [Table 4]). Persistent errors with the updated design included primarily those errors that were not able to be addressed after formative usability testing: (1) wrong rate, (2) wrong indication (e.g., did not select hemoglobinopathy for a sickle cell patient), (3) wrong product (e.g., ordered platelets instead of red blood cells).

Table 4 Summative usability results comparison between old and new order set design

	Old design	New design	p-Value
Error rate	26% (11/42)	12% (4/42)	0.04
SUS score	78 (B)	84 (A)	0.15
Time to task completion	113 s	120 s	0.42
Click count	22/scenario	18/scenario	n/a[a]

Abbreviations: n/a, not available; SUS, system usability scale.

^a Click count was done by manual review of the orders set, and counts were not collected during usability testing.

Positive user feedback on the changes included (1) a shift to using clinical indications rather than asking for specific product modifications and (2) redirection to the correct order group within the order set. Negative feedback included (1) the change of mL to units for patients with dosing weight ≥ 20 kg, (2) frustration with not being able to indicate the exact volume in mL in preparing orders for units, (3) limited clinical indications available for different causes of immunosuppression, and (4) lingering provider questions regarding how long a blood product is reserved for a patient and whether it will be available when needed. Postimplementation in the live EHR environment, there were only four tickets submitted, none of which required emergency changes and two of which were questions about where to find the training documents. No complaints outside of tickets were received from ordering physicians and APPs. The low number of tickets, lack of complaints, and lack of errors are not typical for the size and complexity of this project based on the experience of the authors who have worked in multiple health care organizations in either transfusion medicine and/or clinical informatics.

Discussion of Results

A blood product order set design created by a multidisciplinary team and vetted through organization benchmarking and literature reviews was further improved by formative usability testing with frontline clinicians, resulting in blood product orders and a primary order set with significantly reduced errors in simulation. While the original future state design enabled critical changes (e.g., restricting orders in mL for patients > 20 kg dosing weight, replacing specific product attributes with clinical indications for those attributes), additional error reduction was achieved by the incorporation of user feedback from formative usability testing. Similarly, there were slight improvements in usability scores though not statistically significant. The slight increase in time-to-task-completion is likely due to unfamiliarity with the new design.

Usability testing after initial design is a standard for many health care and software products but it is often done in standardized settings with specific tasks for users to complete.[5] [14] While usability testing in such settings can be helpful for the creation of things such as how order sets appear to hospital physicians and APPs, it is less useful for more complex decision-making such as the right amount of blood to order for a pediatric patient or if product modifications are necessary. In addition, physicians and APPs are often making these decisions while actively providing clinical care meaning they are multi-tasking and having interruptions in their work.

The strengths of this study are several. First, performing usability testing prior to implementation helped determine areas of confusion which need to be rectified prior to implementation. Second, allowing participants to self-direct through the scenarios in a clinical area with frequent interruptions (i.e., in situ) is known to lower the threshold for errors[15] [16] that could occur after implementation. Third, in situ usability improves the diversity of representation and increases the number of testing participants compared with traditional office-based usability testing. This is critically important in an environment with pragmatic needs for clinical operations while engaging in research and teaching of physicians with different levels of experience.[14] [17]

Although not all design errors could be removed due to technical limitations, understanding the existing challenges prior to implementation allows for a proactive approach for education regarding the updates. Even though it cannot always detect or address all issues, formative testing is nonetheless a critical step in process redesign and change management.

Limitations

This study is limited to a multi-institutional complex pediatric health care organization with unique considerations and constraints which may not be translatable to other institutions. Scenarios were created to test specific order set workflows that may or may not reflect the type of decision-making around blood product ordering typically performed by that provider. Testing was also limited to one primary blood product order set which does not account for all the different types of blood product ordering and potential errors in decision-making. Finally, some of the learnings described in this study are specific to one institution; however, many of the design choices and learnings from this study are generalizable to other institutions with similar workflow challenges including (1) definition for aliquots, (2) knowing the maximum volume to infuse with unit product orders, (3) lack of one-to-many linkage between large volume prepare and transfuse orders in mL (specific to EHR vendor), and (4) confusion around type of product modifications required for a patient.

Conclusion

While expert design based on medical evidence and the experience of other institutions is essential to provide a framework for change, new changes can have unintended consequences. Consequently, in situ usability testing, while not exhaustive, is critical in detecting usability flaws and allows for rapid, iterative design changes to avoid unintentional new errors and improve the overall quality of design prior to implementation. For those errors detected during usability testing that cannot be corrected, knowledge of the errors allows for a proactive approach in staff education that can be used to support safe blood product ordering.

Clinical Relevance Statement

Pediatric blood orders and order sets with integrated clinical decision support designed by a committee of experts can help to reduce errors in ordering but there are often unintentional new errors that can occur as a result of major design changes. Usability testing can help to identify design changes that may unintentionally contribute to potential errors and allow them to be corrected prior to implementation. For errors that cannot be technically corrected, identification allows for additional targeted education to physicians and APPs prior to implementation.

Multiple-Choice Questions

1. How does in situ usability differ from traditional usability testing?

   • Occurs over video conference

   • Occurs in research facility

   • Occurs in the work environment of the participating user

   • Occurs in simulation laboratory

   Correct Answer: The correct answer is option c. In situ usability reflects the actual work environment which helps to improve the diversity of representation and increase the number of testing participants compared with traditional office-based usability testing.

2. What is a common source of error and confusion for pediatric providers related to blood product ordering?

   • Maximum appropriate volumes to transfuse

   • Product modifications, also known as special processing

   • Which type of blood product to transfuse first

   • a and b

   Correct Answer: The correct answer is option d. Providers have ongoing issues with ordering too much blood to be administered to pediatric patients as well as incorrectly selecting the type of processing for patients. Overtransfusion can be reduced by incorporating weight limits in the ordering sets. By allowing the blood bank and transfusion medicine to determine the product modification needed based on diagnosis indications, it allows the experts to ensure correct processing.

Conflict of Interest

S.T. has nothing to disclose. H.W. has nothing to disclose. D.R. has nothing to disclose. E.O. has nothing to disclose. A.C. has nothing to disclose. M.R. has nothing to disclose. B.R. has nothing to disclose. S.K. has nothing to disclose. E.O. reports others from PhraseHealth outside the submitted work.

Protection of Human and Animal Subjects

This work was felt to be primarily focused on quality improvement and therefore deemed nonhuman subjects research by the Institutional Review Board of Children's Healthcare of Atlanta.

• References

• 1 Bolton-Maggs PHB, Watt A. Transfusion errors—can they be eliminated?. Br J Haematol 2020; 189 (01) 9-20
• 2 Bachowski G, Borge D, Brunker P, Eder A, Fialkow L, Fridey J. A Compendium of Transfusion Practice Guidelines. Washington, DC: American Red Cross; 2017
• 3 Yarahuan JWB, Billet A, Hron JD. A quality improvement initiative to decrease platelet ordering errors and a proposed model for evaluating clinical decision support effectiveness. Appl Clin Inform 2019; 10 (03) 505-512
• 4 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
• 5 Morse B, Anstett T, Mistry N. et al. User-centered design to reduce inappropriate blood transfusion orders. Appl Clin Inform 2023; 14 (01) 28-36
• 6 Schneider T, Jackups Jr R. Transfusion medicine informatics: a review of current practice and a glimpse into the future. Clin Lab Med 2021; 41 (04) 713-725
• 7 Sanderson B, Field JD, Kocaballi AB. et al. Multicenter, multidisciplinary user-centered design of a clinical decision-support and simulation system for massive transfusion. Transfusion 2023; 63 (05) 993-1004
• 8 Sanderson BJ, Field JD, Kocaballi AB. et al. Clinical decision support versus a paper-based protocol for massive transfusion: impact on decision outcomes in a simulation study. Transfusion 2023; 63 (12) 2225-2233
• 9 Erik H. FRAM: The Functional Resonance Analysis Method: Modelling Complex Socio-Technical Systems. CRC Press; 2017
• 10 Carayon P, Schoofs Hundt A, Karsh BT. et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 (suppl 1, suppl 1): i50-i58
• 11 Kandaswamy S, Jones J, Laporte D, Orenstein E. Combining lean methodology with functional resonance analysis method to understand blood product administration. Proc Hum Factors Ergon Soc Annu Meet 2022; 66: 766-768
• 12 Motulsky SL. Is member checking the gold standard of quality in qualitative research?. Qual Psychol 2021; 8: 389-406
• 13 Brooke J. SUS: a quick and dirty usability scale. Usability Eval Ind 1995; 189-194
• 14 Yen P-Y, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
• 15 Arkley J, Ong LP, Gunaratnam N, Butt T, Clark SC. A distractions capture tool for cardiac surgery and lung transplantation: impact on outcomes. J Cardiothorac Surg 2023; 18 (01) 46
• 16 Liu D, Grundgeiger T, Sanderson PM, Jenkins SA, Leane TA. Interruptions and blood transfusion checks: lessons from the simulated operating room. Anesth Analg 2009; 108 (01) 219-222
• 17 Mann DM, Chokshi SK, Kushniruk A. Bridging the gap between academic research and pragmatic needs in usability: a hybrid approach to usability evaluation of health care information systems. JMIR Human Factors 2018; 5 (04) e10721

Publication History

Received: 03 November 2023

Accepted: 24 June 2024

Accepted Manuscript online:
25 June 2024

Article published online:
25 September 2024