Keywords user-centered design - think aloud - nudge - EHR - user experience - transfusion
Background and Significance
Background and Significance
Clinical decision support (CDS) tools help improve patient safety and reduce variability
in care. Still, there are limitations to their use[1 ] and some questions remain about the best ways to implement them. User-centered design
(UCD) integrates the user in the solution design process,[2 ] using their direct feedback to optimize the design of an intervention. Some benefits
of implementing a UCD approach include gaining buy-in from intended users, domain
knowledge that informs the design, and tacit knowledge that users bring to the table
in the design process.[3 ]
[4 ]
At our institution, we are studying the effects of different CDS designs on quality
outcome measures, specifically on blood transfusion rates. Though a life-saving treatment
for some patients, transfusions of blood products can result in patient harm and excess
costs for health systems[5 ]
[6 ]
[7 ]
[8 ]
[9 ] when used indiscriminately. Optimal blood product utilization requires a balance
between clinical benefit, costs, and risks associated with transfusions, which is
why guidelines strongly recommend a “restrictive” RBC transfusion threshold of a hemoglobin
(Hgb) of 7 g/dL for most hospitalized adult patients,[5 ]
[10 ]
[11 ] although compliance with transfusions guidelines is poor.[12 ] Based on others' successful quality improvement (QI) projects,[13 ]
[14 ]
[15 ] we created a CDS system targeted to computerized order entry to curb guideline-discordant
transfusions at our institution.
This report details how we used UCD to build three distinct CDS designs. In contrast
to previous studies that tested a single redesigned CDS in the electronic health record
(EHR), we set out to study the comparative effectiveness of distinctive designs. We
are deploying these three versions in a randomized fashion. Those results will be
reported in a separate manuscript.
Objectives
The objective of this study was to integrate users' insights on the development of
three different blood transfusion orders and understand how users experienced all
three designs, identify pain points, and streamline the workflow. This allowed us
to ensure the experimental arm designs were optimized for users and distinctive for
comparison.
Methods
This manuscript focused on our use of UCD to optimize the three different versions
of our intervention as follows: (1) version A: No Alert, (2) version B: Noninterruptive
Alert, and (3) version C: Interruptive Alert. The No Alert version included only the
general improvements described below but did not include any alerts. The Noninterruptive
Alert version included a noninterruptive text alert detailing evidence-based transfusion
recommendations that appeared if the patient's most recent hemoglobin level was above
6.9 g/dL. Finally, the Interruptive Alert version also included an alert detailing
evidence-based transfusion recommendations that appeared if the patient's most recent
hemoglobin level was above 6.9 g/dL but, in contrast to the noninterruptive alert,
this version had an interruptive “pop-up” alert that appeared when the user chose
to transfuse patients. This alert offered users the option to remove the order which
resulted in no blood product being ordered. Alternatively, users could continue to
order blood and were asked to select the reason for proceeding with the intended order
with the selections reflective of the guideline indications for blood transfusions.
In addition to different alert mechanisms, we also made general improvements (from
baseline) to each version of the intervention. At our institution, transfusing blood
requires two orders from the clinician: prepare order and transfuse order. The prepare
order directs the blood bank to prepare the unit of blood for transfusion. The transfuse
order instructs the nurse to administer the blood, once received from the Blood Bank.
The general improvements incorporated into each of the intervention versions include
changes to both the prepare and transfuse orders ([Figs. 1 ] and [2 ]). Changes to the prepare order included removing the indications and the multiunit
prepare buttons. Changes to the transfuse order included removing the multi-unit transfusion
buttons and adding guideline-concordant transfusion indication options. These changes
used behavioral nudges to guide users to transfuse according to guidelines. A behavioral
nudge is a minor change in framing choice that predictively alters people's behavior.[16 ] These changes were intended to be more intuitive for ordering clinicians and to
reduce the number of clicks, decisions, and overall cognitive load. Additionally,
by automatically choosing one unit to transfuse, eliminating readily available options
for multi-unit transfusions (adding an extra step to transfuse more than one unit
at a time), and forcing clinicians to choose from a list of guideline-concordant indications,
these changes incorporate behavioral nudges to encourage clinicians to order within
guidelines. Using behavioral nudges was a key guiding principle for this redesign.
Fig. 1 Prepare order – original and UCD final version. Screenshots of the original and tested/final versions of the Prepare Order with changes
detailed including removing the multiunit options for selection, defaulting the selection
to one unit, and removing the nonguideline-based indications. UCD, user-centered design.
Fig. 2 Transfuse order—original and UCD final version. Original and UCD tested/final version (all arms). Screenshots of the original and
tested/final versions of the transfuse order with changes detailed including removing
the multi-unit options for selection, defaulting the selection to 1 unit, and adding
guideline-based indications. Note: in our final version implemented in our eventual
trial, the indications only appear if the pre-Hgb is less than 7.0 g/dL, a change
suggested by user in UCD testing. Hgb, hemoglobin; UCD, user-centered design.
The setting was the University of Colorado Hospital, an urban academic medical center
with approximately 100,000 units of nonoperative units of blood ordered annually.
We recruited 14 clinicians to interact with the orders while keeping a clinical vignette
in mind, allowing the users to interact with the orders in a realistic setting and
provide feedback on order usability and user experience.
We tested all three alternative versions (A, B, and C) of the new orders described
above using the think aloud approach. During think aloud assessment, we conducted
an integrated semi-structured interview for additional feedback. Additionally, six
specific metrics were coded during the interaction to inform the preferences of our
users that we described with descriptive statistics.
Subjects and Recruitment
We recruited 14 clinicians who consistently utilized the EHR (Epic Systems Corporation,
Verona Wisconsin, United States) blood transfusion orders in their routine practice.
We excluded clinicians that do not frequently place blood transfusion orders. Potential
participants were identified by research project staff. We recruited participants
by email invitation. We extended invitations to additional clinicians as suggested
by participants. A purposive sampling framework[17 ] was used to ensure experienced-based insights on the different order designs.
We invited a diverse set of research participants to ensure a dissimilar set of perspectives
and user needs, including participants from Emergency Medicine, Bone Marrow Transplant,
General Surgery, Hospital Medicine, Surgical ICU, Medical ICU, Medicine subspecialists,
and Orthopedic surgery ([Table 1 ]). Recruitment continued until enough clinicians agreed to be in the study to reach
qualitative observational saturation in the sample. The goal was to have enough participants
interact with the orders to get a breadth of insight and continue until we did not
hear or observe new perspectives during the think aloud protocol. Participants were
not compensated for their participation in this study. Our local IRB designated this
as a QI effort, nonhuman subject research.
Table 1
Characteristics of study participants
Clinical service
Medicine
7 (50%)
Surgery
5 (36%)
Emergency medicine
2 (14%)
Clinical role
Attending
3 (21%)
APP
7 (50%)
Resident/fellow
4 (29%)
Gender
Male
7 (50%)
Female
7 (50%)
Abbreviation: APP, advanced practice provider.
User-Centered Design Sessions
The UCD sessions were approximately 45 minutes each, conducted virtually using Zoom
video conferencing. We first communicated to the research participants that the work
being conducted was considered QI and that no consent was needed. We did not collect
any personal health information during the UCD sessions. Each participant was asked
if they would give their permission for the session to be recorded; all participants
provided permission for the session to be recorded. The goals of the session were
described to each of the participants. We outlined the structure of the session, including
the current state and exposure to three different test versions of the blood transfusion
orders without providing details of the changes.
Before the think aloud began, we instructed participants to verbalize their decision-making
and their thought process in line with the think aloud methodology.[18 ] The think aloud methodology requires participants to verbalize their cognitive decision-making
as they moved through the task. This approach helps identify affordances, hindrances,
and pain points of the usability of the task's user interface. If participants proceeded
without thinking aloud, we prompted participants to pause and verbalize their thought
processes so we could collect their insights.
We first instructed participants to read a prepared clinical vignette inspired by
actual cases from the teams' clinical experience. The case was specifically written
to direct participants to order a blood transfusion against generally accepted clinical
guidelines. Participants were asked to keep this clinical case in mind when placing
orders in each of the provided versions. Participants were then asked to place orders
using the current order to ensure maximal proximal familiarity. We then directed users
to carry out the same task in each of the three experimental arms. We asked clinicians
about order navigation, layout, specific content, usability, and how the designs flowed
in comparison to other common orders used. This structured interaction took place
four times in the span of approximately 45 minutes. This amount of time was suitable
for the evaluation and allowed each participant to share their opinions, both positive
and negative, to refine the designs of each of the orders based on expert opinion.
Qualitative Analysis
Transcripts collected from the blood transfusion reduction UCD sessions were prepared
by a professional transcriptionist. Transcripts were de-identified, time-stamped,
and compared to recordings to correct any mistakes.
Importantly, we used a rapid analysis approach,[19 ] an efficient and flexible approach that is a good match for QI work in the health
sciences due to the time it saves in analyzing qualitative data to arrive at an understanding
of the data set. This technique allows rapid, iterative analysis of micro-interactions
like the experience users have with a blood transfusion order. It is also characterized
by the fluidity of data analysis, which occurs simultaneously with data collection,
as well as a positivist and explanatory frame. The transcripts from the UCD think
aloud sessions were the unit of analysis in this work, and the sample was 14 think
aloud sessions.
In this approach, a data entry form is created for both the think aloud session and
the semi-structured interview, giving each question or section a neutral domain name.
Instead of the conventional coding of thematic qualitative analysis, researchers enter
observations in real-time during the UCD exercise into a domain identified in the
protocol and/or interview guide. These templates are shared amongst the research team
so that consistency in observational findings can be confirmed. Findings from observations
can be used to iteratively inform the UCD sessions. The observations, once there has
been observational saturation, informed the recommendations for the design of the
different blood transfusion orders.
Results
The order design with an interruptive alert (version C) was the most preferred version,
with 8 out of a total of 14 users (57%) indicating it as “most preferred.” Version
A (general improvements) and version B (general improvements plus in-line noninterruptive
nudges) were each rated as “most preferred” by 3 of 14 users (21.5%). User comments
suggested that their preference was based on the design's effectiveness at warning
them that they were ordering a nonindicated therapy. 100% of users noticed the new
nudges in the order design that included an interruptive alert, compared with only
57% of users in the design in which the nudges were incorporated into the order as
in-line text. Users reported the interruptive alert was better at getting the user's
attention compared with in-line text, although they also commented that after repeated
exposures to the interruptive alert, it might be a more cumbersome imposition on their
work than the in-line text, especially once the expected behavior (e.g., only transfusing
blood for a hemoglobin of <7.0 mg/dL) had been learned. They also expressed concern
that in emergencies where blood is needed expediently, interruptive alerts add time
to the workflow and might adversely affect patient care. Exemplar quotes are provided
for versions A, B, and C in [Tables 2 ], [3 ], and [4 ].
Table 2
Exemplar quotes related to version A
Nudge to 1 unit
Okay. I think that will be okay. It will probably require some education because I
don't want people to think they're transfusing two units when they click prepare for
transfusion of two units and then have this default to one without them like catching
it. So I think with some education that like, if you're transfusing more than one,
you have to change it. That will be fine. But because it is auto selected and it doesn't
make you select it, you might just skip right over that and not catch that. You're
not transfusing as much as you thought you were (PID: BTR004).
I mean, I think defaulting to one unit is a good idea because majority of the time,
we're writing one unit at a time. (PID: BTR001).
We take care of trauma patients. We take care of transplant patients and it's not
uncommon for us to give three, six units in a few hours at times. We prefer not to
use massive transfusion protocol when we can especially in transplant patients with
immunosuppression antibodies, things like that (PID: BTR006).
Well, certainly from our perspective, it is our routine and built into our order sets
and everything to transfuse one unit, which is reasonable in most cases. And this
is something you can change, I guess. But defaulting, I think you'll save blood because
many times people will say, “Let's give two units,” and that used to be our standard
until we switched a while back. (PID: BTR014).
Table 3
Exemplar quotes related to Version B
Inline guidelines
Yeah. But in an ideal world, it would be like the previous encounter where it was
embedded in the “transfuse”. So then you're only filling it out once. (PID: BTR011)
Well, I don't mind the text. I think this is good info, especially because we're at
a teaching hospital and there's a lot of people who are sort of new in their careers,
or they get flustered, or they don't know exactly what the guidelines are. So, I think
this is helpful. And so in my mind, once I've seen this, I would probably scroll through
because it's kind of redundant down here. And so I don't know if this is worth just
popping up as an interruptive alert, or if just staying here, because it's just one
more thing you have to scroll through maybe. But it's not bad. It's just more words.
(PID: BTR012)
Table 4
Exemplar quotes related to Version C
Interruptive alert guidelines
I mean, I think probably the interruptive alert would probably be the best for pointing
out that you're ordering something that's not indicated. It's pretty easy to look
past that, like, that help text, especially when if it's not in bright red. (PID BTR001)
Prob, so an interruptive alert always makes me pause and at least think for a second.
(PID: BTR008).
I click the transfuse RBC, and I'm acknowledging the reason, and it's this one here.
And I say, okay. But then I have to click it again. Is there a way to make it, so
that it populates—whatever I say—because these are the same selections that I previously
had to click. But is there any way to carry it over so that it populates into this
box? (PID: BTR013)
I think ideally having both the prepare and transfuse, the interruptive alert having
to click the indication twice was a little more cumbersome and slows things down.
A lot of times when we need blood, we need it quickly. And so adding steps or significant
areas or places to miss something like actually transfusing and then having to wait
for the nurse to try to release it, the blood bank to call and say they don't have
a transfuse order can be detrimental. (PID: BTR006).
I click the transfuse RBC, and I'm acknowledging the reason, and it's this one here.
And I say, okay. But then I have to click it again. Is there a way to make it so that
it populates—whatever I say—because these are the same selections that I previously
had to click. But is there any way to carry it over so that it populates into this
box? (PID: BTR013)
To make the in-line text more noticeable, many users agreed that larger font size
in a more noticeable color such as red would add visibility ([Fig. 3 ]). Also, while the in-line text was customized to their patient's hemoglobin value,
most users (89%) did not realize this and thought the in-line text was a generic recommendation.
Regarding the interruptive alert, users agreed that it would compromise their workflow
less if it allowed users to change their order within the interruptive alert itself,
rather than only making suggestions which they would then need to return to the order
to correct ([Fig. 4 ]).
Fig. 3 Arm B—noninterruptive—before and after UCD testing. Screenshots of the before and after UCD testing with changes to noninterruptive alert
detailed including adding it to the order, not just the transfuse order, making alert
text larger and in orange to be more visible, adding last Hgb result, adding recommendation
for 1U PRBC transfusion at a time in bed. We also reduced the amount of alert text
in the Transfuse Order. Hgb, hemoglobin; UCD, user-centered design.
Fig. 4 Arm C—interruptive alert—before and after UCD testing. Screenshots of the before and after UCD testing with changes to the interruptive
alert detailed including elevating lab result to higher in the view, removing indications
text, adding indications as buttons, and requiring comment for “Other” selection.
UCD, user-centered design.
The general improvements to the order, including the rearrangement of some order buttons,
display of the last active Type and Screen (a necessary test to obtain before a blood
transfusion), and updated text regarding guideline-based transfusion recommendations,
were universally endorsed.
Finally, on the initial pass through the new orders, multiple users missed needing
to click the “Transfuse” order to complete the transfusion. When asked, users were
not used to this change and expressed worry that it could be missed. For example,
one user stated, “If I actually wanted to transfuse them and I was moving fast, I
would not see (the transfuse order) …I would ignore it and it wouldn't be until half
an hour down the road that somebody would tell me, hey, you don't have the order in
for transfuse RBCs.”
Discussion
Findings
Users provided valuable feedback on the design of new orders that we incorporated
into the final versions for the subsequent randomized trial. End users have proven
effective in providing insights on transfusion practices[20 ]
[21 ] which was demonstrated here as well. Specifically, users provided comments contrasting
the visual noticeableness of the interruptive and noninterruptive alerts, recommendations
for fewer clicks and more intuitive design, requests to make selections pertinent
to the patient data,[13 ] and finally, implications for missing critical components of the order that may
lead to patient safety concerns. Users also provided feedback on their preference
for different versions.
Critically important for the planned randomized trial was that we found a large discrepancy
in the number of users who noticed the in-line text compared to those who noticed
the interruptive alert. With this feedback, we adjusted the in-line text display and
the interruptive text display ([Fig. 2 ]) so that we could truly evaluate “interruption” and not “did you even see the words.”
This is a critical point for the future study of CDS systems. Our approach reinforced
that first-pass UCD and usability testing are important to make sure that investigators
are studying what they intend, in this case, the “interruptive” versus “noninterruptive”
properties of the CDS system and are not confounded by differences in human factors
engineering choices that may hamper the effectiveness of one design over the other.
Regarding making the orders more user-friendly and less burdensome, users recommended
less text, making patient-specific data more visible, bolding recommended actions,
reducing the number of clicks, and making the flow of actions more intuitive. We incorporated
this feedback by moving the patient-specific lab data about the last resulted Hgb,
bolding the recommendation to transfuse one unit of packed red blood cells at a time,
and removing text ([Fig. 2 ]). We made the flow of actions more intuitive particularly in the interruptive alert
by changing the workflow. The result was that users could select the indication for
transfusion directly in the alert removing the need to duplicate the indication in
the transfusion order ([Fig. 3 ]).
To respond to users' recommendations to make the orders more specific and relevant
to the individual patient, across each of the arms of the subsequent trial, we made
the indications for transfusion specific to the patient data. Specifically, this meant
incorporating display logic such that if the patient had a last known hemoglobin level
of 6.9 g/dL or less, the indications no longer are displayed, and users may proceed
([Fig. 2 ]). This was done because one of the generally accepted blood transfusion indications
is a hemoglobin level less than 7.0 g/dL. Thus, a patient with a hemoglobin level
less than 7.0 g/dL already has an accepted indication so no other justification for
the transfusion is needed. This is in line with the “nudge” methodology to make doing
the right thing (transfusing for an accepted recommendation) easier.[16 ]
[22 ] One important notation here is that before receiving this feedback from users, this
was not a design feature we considered or even knew was possible. Our users directed
us to ask questions of the EHR platform that we would not have otherwise known were
possible.
Lastly, we made a design choice to not precheck the transfusion order, only to prepare
PRBC order. We hypothesized this would make users consider transfusing more carefully
if a click was required. However, in the user-centered think aloud, we noticed multiple
users did not notice the transfuse order needed to be clicked and thought they had
successfully ordered the blood transfusion. This presents a potential patient safety
issue if patients are not receiving blood transfusions when the clinicians intended.
Thus, based on the results of these sessions we elected to precheck the transfuse
order in the final versions, potentially reducing user error as has been done in several
recent studies,[14 ]
[23 ] incorporated into the subsequent randomized trial.
Many of the above recommendations provided by users through these sessions reinforced
our governing principle of using behavioral nudges, which is supported by previous
work.[24 ]
[25 ]
[26 ] One of the strongest nudges, the display logic for the transfusion indications,
came at the suggestion of the users who participated. Before these sessions, we did
not consider this feature and did not know it was technically possible. At the urging
of our users, we inquired with the EHR analysts and discovered display logic could
be integrated into the orders.
Regarding the type of alerts, we were surprised that most users preferred the interruptive
alert over the noninterruptive alerts, based on prior evidence that noninterruptive
alerts are usually preferred by clinicians.[1 ] However, the users offered several caveats to this endorsement suggesting that the
preference would wane with time, especially once the desired behavior was learned.
Thus, the interruptive alert might progress from “helpful” to “annoying.”
Without attention to human factors and design, the failure or success of any of the
versions of the order might be confounded by the failure of the design, for example
by failing to draw the user's attention to the most salient elements. We wanted the
changes to our orders to be maximally visible to users in the clinical environment
from the outset, ensuring the results of our randomized study were purely due to these
changes and no other human factors. We, therefore, used a UCD approach and incorporated
end-user insights and suggestions collected during a simulated clinical scenario within
an EHR test environment to refine and optimize the order design before the implementation
of the larger CDS study.
Limitations
Our study has several limitations to generalizability. It was only performed at one
institution, it focused on only one CDS system for one clinical scenario, and most
participants were advanced practice providers.
We had 14 participants, though 8 to 10 participants were found to be sufficient for
usability testing.[27 ] While we sought to draw from a broad range of specialties that typically order blood
transfusions, there is known variability in transfusion practice, and therefore possibly
in order design preference, between subspecialties, not all of which were represented
in our sample. This may also limit the generalizability of our results.
Subjects' responses may have been influenced by their awareness that they were in
a usability study of a CDS tool so their responses to differences in design may be
different in actual clinical practice. The results of our randomized trial of the
different order versions will be gathered in a real-world clinical scenario and may
illuminate this possible discrepancy. Finally, the clinical vignette, though written
to be applicable across specialties, may not have been representative of all clinical
scenarios participants may encounter. Further, the vignette instructed participants
to transfuse against generally accepted guidelines which, though common in clinical
practice, may have limited the “realness” of the scenario depending on the practice
patterns and guideline adherence of the participants.
Conclusion
In a randomized study of different CDS designs for a QI project aimed at reducing
inappropriate blood transfusions, our approach of implementing a think aloud UCD protocol
to record stakeholder input informed our CDS interventions. This approach may make
it easier to discriminate the effects of these interventions being “interruptive”
and “noninterruptive” by potentially ensuring parity in human factors among the different
versions of the intervention. We feel confident that by conducting UCD sessions, we
added value to the subsequent randomized trial by producing end-user informed final
versions of the displays. Future investigators should incorporate user-centered approaches
to inform CDS clinical trials.
Clinical Relevance Statement
Clinical Relevance Statement
This study can be used as a guide for the design of EHR order templates to improve
workflow, adherence to guidelines, or in preparation for a clinical trial. The frequency
of order use and perceptions of usability are important human factor considerations.
User-centered design of orders in the EHR can improve clinician satisfaction, reduce
burnout, and improve the perception of EHR usefulness in clinical care.
Multiple Choice Questions
Multiple Choice Questions
What version of the blood transfusion order screens incorporated nudge principles
to steer user toward guideline concordant ordering?
What is the definition of a behavioral nudge?
Restriction of choices such that only the desired options are available.
Minor change in framing choice that predictively alters people's behavior.
Written text that provides an explanation for each choice presented to users.
Significant economic reimbursement for desired outcomes.
Correct Answer: The correct answer is option b. A behavioral nudge is a minor change
in framing choice that predictively alters people's behavior.
