Keywords
medication management - clinical decision support systems - graphical user interface
- interfaces and usability - requirements analysis and design
Background and Significance
Background and Significance
Nitroglycerin is a vasoactive medication for acute relief of chest pain.[1] There is no standardized dose of intravenous nitroglycerin because the medication's
impact on blood pressure varies between patients.[2] While chest pain relief may be achieved for some patients at a low nitroglycerin
dose, others require far greater doses to achieve the same outcome. Therefore, a trained
nurse must manually increase or decrease the infusion (called titration) to achieve
the desired dose.[2]
[3] If the dose is not optimal, patients can experience side effects such as dizziness,
light-headedness, nausea, vomiting, palpitations, headaches, and reflex tachycardia.[2] Adjusting the dosage is ultimately a trial-and-error process, and titrating incorrectly
is common. In a descriptive study, only 8% of vasoactive titration events among 60
critical care patients were deemed titrated correctly.[4]
In our previous work, we developed a Nitroglycerin Dose Titration Decision Support System (nitro DSS) using data from the eICU database developed by Philips Healthcare. Specifically,
we determined the accuracy of the machine learning algorithms used in nitro DSS to predict near-future blood pressure measurements for patients receiving nitroglycerin
infusions.[5] A clinical decision support system (CDSS) helps in the decision-making related to
patient care through targeted patient information, clinical knowledge, and relevant
data.[6] CDSS applies data to algorithms and provides decision options based on the patient's
specific circumstances.[7] The potential options are promptly delivered to clinicians. The nitro DSS predicts blood pressure to a potential new dose change of nitroglycerin, which could
aid nurses' decision making for titrating nitroglycerin infusions. A nurse can input
the prospective new dosage value into the system, and a predicted blood pressure would
be generated. Based on this output and the nurse's clinical judgment, they can adjust
the dose.[7] Consider a patient receiving a nitroglycerin infusion at 15 μg/min, who has ongoing
chest pain rated at 10/10, and whose blood pressure is 130/80 mm Hg. The nurse considers
increasing the dose to 20 μg/min to relieve the chest pain. The nurse would enter
the potential new dose of 20 μg/min into the interface, and the predicted blood pressure
would populate reading 128/78 mm Hg. The nurse can review the predicted blood pressure
and apply their clinical judgment to decide whether to accept or reenter a different
dose. CDSS developed for nursing tasks are designed to enhance nurses' decision-making.
The nitro DSS requires a user interface (visual display of the system) to enable interaction between
nurses and the system. To design the interface, it is essential to focus on design
features that make the system easy for the nurse to use.[8] Prior research has indicated that CDSS with critical design limitations, which were
challenging to use, resulted in mistrust of the recommendations, leading to a low
usage rate and increased user frustration.[9] These issues limit the potential utility of highly accurate predictive tools. For
this reason, optimizing the design of the user interface is crucial for successful
adoption.[10]
User-centered design (UCD) is a widely used approach that grounds the product design
process on the needs and understanding of users.[11] UCD involves user engagement through interviews, surveys, and task analysis throughout
the design process. Multiple iterative cycles of usability testing, which consists
of a group of end users testing the prototype and providing feedback on the interface
design, are usually performed to improve the interface design.[12]
[13]
[14] Several studies integrating a UCD approach in developing health technology products
have shown interventions to be well-used and effective when tested.[15]
[16]
[17]
[18]
Objective
The objective of this study is to design the nitro DSS user interface for titrating intravenous nitroglycerin doses through UCD principles.
Methods
To design the nitro DSS interface, a UCD approach consisting of two phases was conducted: (1) a qualitative
study to identify design specifications for prototype development and (2) usability
testing. In both phases, a demographic questionnaire was completed.
Qualitative Study to Identify Design Specifications for Prototype Development
The objective of the qualitative study phase was to understand the coronary care nurses'
needs and preferences for the nitro DSS interface and translate those into initial design specifications. To achieve this
objective, the interview session focused on two sections: (1) the cognitive task of
titrating intravenous nitroglycerin infusions and (2) the interface design specifications.
Cognitive task analysis involves uncovering and representing users' cognitive activities
to perform specific tasks.[19] This technique was used to unpack the cognitive elements underlying nurses' decision-making
when titrating nitroglycerin infusions. Understanding the nurses' cognitive process
and decision-making can assist in designing the interface by ensuring all information
and functions needed to titrate decision-making are included.[13]
[19] Nurses were also asked to share their perceived wants and needs of nitro DSS while accounting for the identified constraint that it would be integrated into an
existing system that contains blood pressure and nitroglycerin titration data (i.e.,
an electronic health record [EHR]), as this is a feasible implementation approach.
The semistructured interview guide was developed based on cognitive task analysis
and literature about user interface design. Questions were extracted from an interview
guide on exploring the decision-making of critical care nurses in performing titration
of vasoactive drugs in cardiac patients for the cognitive task analysis.[20] The second section of the interview guide (interface design specifications) was
developed to reflect the design and functional specifications for the decision support
system.[21] All sessions of this study phase were virtual using Microsoft Teams at a time and
date convenient for the participants. Interviews were audio-recorded.
The results of this phase identified initial design specifications for the nitro DSS interface, which were used to build the interface prototype using Figma, a tool for
building interactive data-focused products.[22]
Usability Testing
Three in-person iterative usability testing cycles were conducted to test the prototype,
and qualitative and quantitative data were collected. The results from each usability
testing cycle are used to refine the interface and retest in the next cycle until
prototype refinement is achieved (typically two to three cycles).[13] The UCD sessions were approximately 45 minutes each and held at the hospital at
a convenient time and date for nurses. In each iterative usability testing cycle,
the prototype was presented to the nurses on a laptop, and a brief overview of the
functions of the developed nitro DSS was provided. Nurses then had the opportunity to interact with the prototype by entering
potential new doses for which blood pressure predictions were generated.
In each round of usability testing, three titration examples were provided (see [Table 1]). Examples were intentionally created to gather insight into situations where nurses
might consider predictions implausible given the current blood pressure and dose change.
For instance, in round 2, nurses were instructed to enter a dose increase from 15
to 20 μg/min. The initial blood pressure was 134/77 mm Hg, and the predicted blood
pressure with the dose increase was 175/88 mm Hg. In round 3, nurses were again presented
with the blood pressure prediction of 175/88 mm Hg from the initial 90/60 mm Hg but
for a dose change from 30 to 20 μg/min. Nurses were instructed to “think aloud” by
expressing their thoughts, feelings, and opinions while interacting with the prototype.
This approach is ideal for evaluating a prototype in the initial development stages
because major usability problems can be identified with small samples.[19] Field notes were taken during the usability testing sessions.
Table 1
Actual and predicted blood pressure data for potential nitroglycerin dose (μg/min)
of usability testing rounds
|
Example 1
|
Example 2
|
Example 3
|
|
Testing round
|
Potential dose (μg/min)
|
5
|
10
|
15
|
20[a]
|
30[a]
|
20[a]
|
|
Round 1
|
Predicted blood pressure (mm Hg)
|
134/77
|
127/74
|
133/78
|
133/73
|
89/58
|
109/62
|
|
Actual blood pressure (mm Hg)
|
144/80
|
128/78
|
134/77
|
133/73
|
90/60
|
110/60
|
|
Round 2
|
Predicted blood pressure (mm Hg)
|
134/77
|
127/74
|
133/78
|
175/88
|
89/58
|
109/62
|
|
Actual blood pressure (mm Hg)
|
144/80
|
128/78
|
134/77
|
133/73
|
90/60
|
110/60
|
|
Round 3
|
Predicted blood pressure (mm Hg)
|
134/77
|
127/74
|
133/78
|
133/73
|
89/58
|
175/88
|
|
Actual blood pressure (mm Hg)
|
144/80
|
128/78
|
134/77
|
133/73
|
90/60
|
110/60
|
a Indicates the doses nurses entered into the Nitroglycerin Dose Titration Decision Support System prototype. Data from doses 5 to 15 μg/min was prepopulated on the interface.
Once the nurses tested the prototype, they were asked to complete quantitative measures
consisting of two brief surveys: (1) The System Usability Scale (SUS) and (2) the
Ottawa Acceptability of Decision Rules Instrument (OADRI). The SUS is widely used
to evaluate the subjective usability of newly developed devices and systems.[23]
[24] Usability is the extent to which a product can be used to achieve specified goals
in a particular context.[25] The SUS is a simple, 10-item scale that yields valid and reliable scores among respondents
in professional contexts. It has five response options ranging from strongly agree
to strongly disagree.[24] This scale has garnered extensive use in assessing usability relating to health
care applications.[26]
[27]
[28] OADRI is a validated instrument to measure the acceptability of clinical decision
rules in the context of CDSS; therefore, it was selected for use in this study.[29] It is a 12-item instrument on a 6-point Likert scale (1 = strongly disagree, 6 = strongly
agree). A neutral midpoint was included in the response options, allowing respondents
to select no option/do not know. OADRI groups various barriers and facilitators related
to whether a research innovation will be adopted into practice into three large categories:
(1) aspects of the innovation: items 1 to 4, (2) decision-making: items 6 to 9 and
11, and (3) environment items 5, 10, and 12.[30] These categories are captured through questions such as: Is it easy to use, and
is it useful in my practice? Do my colleagues support the use? Does the environment
I work in make it difficult to use?[30] When it was developed, OADRI proved reliable (Cronbach's alpha >0.8). Assessing
acceptability is a fundamental component of a UCD approach.[31] SUS and OADRI are brief questionnaires requiring less than 2 minutes each to complete.
Upon completing the questionnaires, a brief semistructured interview was conducted
to elicit feedback on the nitro DSS interface and ways to improve the prototype (such as what nurses liked and disliked,
how to improve, and how easy it was to use the interface). After completing the first
cycle of usability testing, the results were used to refine the design and retest
it in the second cycle. This process was repeated for a third cycle. Each cycle was
completed when no new usability issues or changes were identified.[12]
Sample, Setting, and Ethical Approval
Convenience sampling was used to recruit nurses from adult coronary care units of
two hospital sites from a single organization in southwestern Ontario, Canada. Participation
in this study was extended to eligible registered nurses who met the following criteria:
(1) currently employed in the coronary care unit and (2) had experience administering
and managing intravenous nitroglycerin infusion and blood pressure monitoring. Nurses
were permitted to participate in both phases of this study. However, efforts were
taken to recruit a new group of nurses for usability testing. Ethical approval was
obtained from the hospital's research ethics board and educational institution.
Data Analysis
A conventional qualitative content analysis approach was used for the interview data.[32] Analysis began after the first interview was completed. All data (summary notes
and audio recordings from the session) were transcribed verbatim and checked for accuracy
before analysis. Data were read word-for-word to derive codes.[33] The codes were organized into categories based on how different codes were related
and linked. The categories were grouped into meaningful themes that identified the
overarching design specifications of nitro DSS—NVivo (Version 12.7) assisted with the data analysis process.[34] Two researchers came together to confirm the codes and themes (investigator triangulation),
ensuring multiple perspectives were incorporated during the analysis. Based on the
results of the qualitative study, a digital interface prototype of the nitro DSS was developed.
Questionnaire scores from the SUS and OADRI were imported into Excel. Descriptive
statistics were used to summarize scores from the SUS and OADRI. For the SUS questionnaire,
the scores were converted to a range from 0 to 100, with 68 representing the minimally
acceptable usability score.[24] The literature identifies a score of 68 as a useful benchmark (mean SUS score),
where 50% of apps fall below and above this threshold.[24]
[35]
[36]
[37] For the OADRI, items rated 1, 2, or 3 were rescaled to 0, 1, and 2, respectively.
Responses “no opinion/do not know” were coded as the middle of the scale (score 3),
as done in previous research.[29]
[38] The responses were reversed for items 8 to 12, so higher scores always represent
higher acceptability. The overall acceptability was calculated by averaging the score
of all 12 items, resulting in a score that ranged from 0 (lowest) to 6 (highest) acceptability,
with an overall mean score >3 indicating good acceptability.[29] A plot of the mean SUS and OARDI scores over the three rounds was created to compare
the usability testing cycles visually.
Results
Study Participants
A total of 20 nurses participated in this study, with 7 nurses for the qualitative
study phase and 15 for the usability testing phase (5 in each of the three cycles).
Two nurses participated in both phases of this study. Most of the nurses were female
(86%), ranging from 25 to 34 years of age (46%). The average experience as a coronary
care nurse was approximately 10 years (ranging from 1 to 26 years), with most nurses
having a bachelor's degree (86%) and the remaining having a diploma (14%). The average
experience with CDSSs was 2.8 years (range 0.6–10 years), and they rated themselves
as competent (41%), proficient (36%), or expert (18%) in using CDSSs. A summary of
participant characteristics is presented in [Table 2].
Table 2
Participant demographics and characteristics
|
Characteristics
|
Total participants (n = 22)
|
Phase 1 participants (n = 7)
|
Phase 2 participants (n = 15)
|
|
Age (y), n (%)
|
|
25–34
|
10 (46%)
|
2 (28%)
|
8 (53%)
|
|
35–44
|
8 (36%)
|
4 (57%)
|
4 (27%)
|
|
45–54
|
2 (9%)
|
1 (14%)
|
1 (7%)
|
|
55–64
|
2 (9%)
|
0
|
2 (13
|
|
Female, n (%)
|
19 (86%)
|
7 (100%)
|
12 (80%)
|
|
Highest level of education completed, n (%)
|
|
Diploma
|
3 (14%)
|
2 (29%)
|
1 (7%)
|
|
Bachelor
|
19 (86%)
|
5 (71%)
|
14 (93%)
|
|
Work experience (y), mean (range)
|
9.8 (1–26)
|
11 (1–26)
|
9.2 (2–23)
|
|
Frequency of nitroglycerin administration/management, n (%)
|
|
Never
|
0
|
0
|
0
|
|
Rarely
|
11 (50%)
|
4 (57%)
|
7 (47%)
|
|
Sometimes
|
9 (41%)
|
3 (43%)
|
6 (40%)
|
|
Often
|
2 (9%)
|
0
|
2 (13%)
|
|
Always
|
0
|
0
|
0
|
|
Proficiency in managing nitroglycerin infusion, n (%)
|
|
Novice
|
0
|
0
|
0
|
|
Advanced beginner
|
3 (14%)
|
0
|
3 (20%)
|
|
Competent
|
12 (55%)
|
4 (57%)
|
8 (53%)
|
|
Proficient
|
6 (27%)
|
3 (43%)
|
3 (20%)
|
|
Expert
|
1 (4%)
|
0
|
1 (7%)
|
|
Experience with CDSS (y), mean (range)
|
2.8 (0.58–10)
|
2.5 (2.5–2.5)
|
3.2 (2.5–10)
|
|
Proficiency using CDSS, n (%)
|
|
Novice
|
0
|
0
|
0
|
|
Advanced beginner
|
1 (5%)
|
1 (14%)
|
0
|
|
Competent
|
9 (41%)
|
1 (14%)
|
8 (53%)
|
|
Proficient
|
8 (36%)
|
5 (72%)
|
3 (20%)
|
|
Expert
|
4 (18%)
|
0
|
4 (27%)
|
Abbreviation: CDSS, clinical decision support system.
Identifying Design Specifications for Prototype Development
Four themes captured the design specifications for the nitro DSS. These included (1) Clear and Consistent Interface, (2) Vigilant, (3) Interoperable,
and (4) Reliability. Exemplar quotes supporting the themes are provided in [Table 3].
Table 3
Exemplar quotes related to the identified themes from the qualitative study phase
|
Theme
|
Quotes
|
|
Clear and consistent interface
|
I think to keep it as simple as possible—white background. I'm thinking keep it the
same or very similar fonts and appearance to the current EHR. So, I think the less
different this new system is from (current EHR), the better, because if it's very
different, we have to take more time to find the information we need. But if it's
designed to be similar and something that we're used to, we know, like your eyes and
your brain processes it easier and faster. (Participant 2)
I would want it to be similar to (current EHR) because it autopopulates under the
medication administration record in (current EHR). (Participant 6)
I want it to be user-friendly. I want the screen to be as simplified as possible.
I don't want too much stuff on it. I want it to be easy to read. I'm keeping in account
that we can see the information easily. It's not very difficult to find or overwhelming.
(Participant 6)
Some patients just aren't quite that straightforward. Sometimes it is good to know
the trend from when they first came in. So, if there was a way to keep track of that
and then have it easily accessible, I don't know if it needs to be on that screen,
but accessible might be helpful. It would be best if you were straightforward, as
that's the important thing, not too many words or too many visuals. I find, especially
in nursing, you're in a hurry to have the information you need without many extras.
(Participant 7)
For the blood pressure trend, it could be something if you hover on the past value
that's already there, that it pops up, and you see the trend graph of how it went.
Otherwise, keep it very simple. (Participant 4)
|
|
Vigilant
|
The Nitroglycerin order is going to come with some parameters. I want an alert when
the predicted blood pressure falls below those parameters. (Participant 2)
You don't want people to ignore it (alert) and you don't want to have it a pop-up
like other pop-ups in (current EHR) where you can't continue charting if you don't
address it. (Participant 3)
If an alert is out of range, alerts are valuable. I think it just needs to be not
too many. It needs to alert us when we need to be aware, but there are so many that
we end up blocking it out. So, if it's a blood pressure we must address, I think having
an alert makes sense. (Participant 7)
|
|
Interoperable
|
It would be best (to integrate) in the MAR. Logically, uptake is best in the MAR because
it will all be done in one place. (Participant 6)
I would want this system to give a predictive blood pressure for a potential dose,
and you click OK, then it flips it into your documentation. I don't want to have to
hit OK and then document. (Participant 1)
|
|
Reliability
|
I want to see how accurate that's (prediction) has been. If it's giving me an answer,
I can trust it because what it told me in the past has been accurate. (Participant
3)
It would be good to know (the performance of the system), and it's nice to have confidence
in the system you're using to know how accurate it is. If I found that whatever titration
I'm doing up or down is not accurate, and I'm making decisions on something that I
don't have confidence in that's going to be accurate or at least somewhat accurately
predict, then I would start to lose confidence (in the system). But then the opposite
would also be true. If I find that the more I use it, the more I find that it's pretty
accurate in predicting, the more confidence I would have in using it. (Participant
7)
It would be too much on the screen, but I don't agree with just blindly trusting.
Could that also be like a hyperlink? If you have suspicions or are wondering about
it, you can look into it, but it won't necessarily interrupt your workflow. (Participant
5)
A graph would be helpful. To see them both drawn out. Like if I could have the option
to have a graph because, at a glance, it would be very helpful over 12 hours not to
go through, you know, every 15 minutes of blood pressure predicted or actual. I would
like to see a graph, which would give me a much quicker snapshot of its accuracy (the
prediction). (Participant 2)
|
Abbreviation: EHR, electronic health record; MAR, medication administration record.
Clear and Consistent Interface
A clear and consistent interface refers to the design and presentation of the interface
in a way that is easily understandable and uniform with existing interfaces, creating
an intuitive experience. Nurses in this study frequently referred to their current
EHR system when asked how the nitro DSS user interface should be designed. They requested that the nitro DSS interface be consistent with their EHR; for example, they asked for the physiological
data to be presented in a flowsheet, as currently shown in the EHR. Consistency with
the EHR makes the nitro DSS interface more straightforward and requires less cognitive load. This was translated
to a design requirement by presenting vital signs, chest pain score, side effects,
and nitroglycerin dosage data in a flowsheet format. Statements from nurses that highlighted
the preference for an interface consistent with the EHR are presented in [Table 3].
The theme of a clear and consistent interface also alludes to the requirement for
the interface to avoid information overload. Nurses emphasized this preference through
repeated descriptions of the burden and adverse impacts of visual overload. They asked
for the user interface of the nitro DSS to present only the information necessary for titration-related decision-making,
and [Supplementary Material] (available in the online version) can be included as hyperlinks or in an “additional
resources” section. Nurses described this as the difference between the “I need to
know” and “good-to-know” information. For example, nurses shared the “need-to-know”
information, including vital signs, dose-related data, and chest pain scores, as this
information is essential to deciding when to increase or decrease the dosage. Nurses
shared that “good-to-know” information, like the nitroglycerin protocol itself, should
be included as a link rather than described on the main page. When asked how they
would want the content to be presented to showcase a clear interface, the nurses said
they wanted to keep it simple and display essential information in a table format.
They declined the need for all vital signs or chest pain scores to be presented in
a graph. Instead, they explained that while the blood pressure graph is valuable information,
it should not be on the main screen and only be accessed when necessary.
Vigilant
A vigilant interface actively monitors patient predictions, prioritizes alerts based
on deviations from normal ranges, and aims to enhance patient safety without burdening
nurses with excessive or unnecessary alerts, thereby addressing the issue of alert
fatigue in health care. Nurses expressed a need for nitro DSS to alert nurses of abnormal predictions that pose a risk to patient safety, but at
the same time, they were wary of the potential impact of excessive alerts on alert
fatigue (see [Table 3]). The nurses claimed they were already overloaded with alerts in their practice.
As such, there was a need for nitro DSS to have a vigilant interface by balancing alert fatigue and patient safety, meaning
a visual alert appears on the screen only when the expected blood pressure deviates
from the desired range, which is defined in nitroglycerin protocol as systolic blood
pressure less than or equal to 90 mm Hg.[39] To reduce fatigue, no auditory alarm was developed.
Interoperable
An interoperable system can exchange data seamlessly and efficiently and interact
with other systems to work together and share information to enable efficiency. Nurses
suggested that the system be integrated into the medication administration record
(MAR) component of the EHR, as this is where nitroglycerin dose changes are documented.
This would allow information on dosing to be automatically entered in the MAR, reducing
double documentation (see [Table 3]). Specifically, they suggested that once a dose is accepted on the nitro DSS interface, this should automatically translate into the MAR, as it will save time
and improve efficiency.
Reliability
Reliability is defined as consistently generating accurate predictions, demonstrating
high trustworthiness in its predictive capabilities, and having transparency in its
performance. Nurses expressed a need to know if the system was reliable. Precisely,
they needed to understand the accuracy of the blood pressure predictions. Knowing
the system was reliable would help nurses develop trust and potentially increase the
use of the nitro DSS. When asked how nurses would like the accuracy of the predicted blood pressure to
be presented, nurses shared that they would like a graph showing the actual and predicted
blood pressure over time (see [Table 3]).
Usability Testing
[Table 4] summarizes the interface development. [Figs. 1] and [2] depict the SUS and OADRI scores collected from the participants throughout the three
usability rounds. [Fig. 3] shows an overview of the prototype versions developed based on the usability testing
results.
Fig. 1 Results of the System Usability Score. Gray bars are individual participant scores.
Orange bars are the mean score for each round of usability testing. Blue bars are
the overall score across all three rounds. A score >68 indicates the above-average
usability of a system. SUS, System Usability Score.
Fig. 2 Results of the Ottawa Acceptability of Decision Rules Instrument score. Gray bars
are individual participant scores. Orange bars are the mean score for each round of
usability testing. Blue bars are the overall score across all three rounds. OADRI,
Ottawa Acceptability of Decision Rules Instrument.
Fig. 3 Overview of the nitro DSS prototypes. Red arrows indicate the changes to the nitroglycerin protocol feature.
The red numeric values in figures from rounds 2 and 3 and the final version represent
abnormal blood pressure measurements. Changes to the side effects row can be seen
in prototypes tested in rounds 3 and 4. nitro DSS, Nitroglycerin Dose Titration Decision Support System.
Table 4
Summary of the iterative development process
|
Round
|
Express user need
|
Design modifications
|
|
Initial prototype
|
Based on the results of Phase 1
• Clearer cues on where to enter the numeric input for new dose
• Clearer cues on where to enter the data entry features placement
• Consistent placement of “Accept dose” or “Cancel dose” options to match the hospital
EMR
• Have the blood pressure graph as an optional feature so users can quickly identify
patient trends
• Need for data on vital signs, chest pain score, and nitroglycerin dosage presented
in a table format with time navigation
• Clear icon use for nitroglycerin protocol
• Nee for an alert when low systolic blood pressure prediction is below the threshold
of 90 mm Hg
|
• Be able to enter a numeric value in the potentially new dose rather than preselected
options. The predict button should be green. The predicted blood pressure should be
in blue font color
• Having all data entry features at the top
• The accept dose or cancel features in the bottom right corner are blue
• Having a blood pressure graph displaying actual versus predicted blood pressure
over time as a feature. But not displayed on the main screen. This would be an option
the user can select to view if needed. This feature can be called Actual
• Having a table that reflects the current flowsheet in the current EHR displaying
the vitals (blood pressure, heart rate, Resp., SpO2), chest pain score, and nitroglycerin dose. The table only shows five timeslots at
a time but can have a slider feature where they can slide to see the different data
in the time columns, that is, I can move to see information at 1,135 but move back
to see 1,100 information.
• Having a “paperclip” icon showing an attachment to indicate nitroglycerin protocol
• Generate an alert for blood pressure predictions of systolic below 90 mm Hg. This
prediction would appear below the predicted blood pressure stating, “The system predicted
a critically low blood pressure.” Have the option to select okay/close the alert
|
|
Round 1 (version 2)
|
• Inclusion of mean arterial pressure in predicted blood pressure
• Removal of irrelevant cues from the flowsheet
• Increased clarity of nitroglycerin protocol feature
• Clearer identification of abnormal values
• Enhanced blood pressure graph readability
• Clarity in feature placement
|
• Add mean arterial pressure beside blood pressure in table
• Delete RR from the flowsheet
• Change the attachment icon to Nitroglycerin Protocol
• Add a nitroglycerin dose graph
• Make abnormal values in red (the blood pressure value of 90/60 in chart in red
as “!90/60” and heart rate to !110 from 110)
• In the graph add numeric values for blood pressure and time, that is, @1100: 144/80
• Make the abnormal blood pressure alert appear in the center of the screen rather
than right under the blood pressure. The rest of the screen should be greyed out and
only the alert is displayed.
• Move blood pressure and predicted blood pressure closer to the center rather than
on the right side.
• Once you press accept dose that option disappears
**Incorporated a scenario to have the dose change from 15μg/min with a blood pressure
of 134/77 mm Hg to 20 μg/min with a predicted blood pressure of 175/88 mm Hg for the
first prediction
|
|
Round 2 (version 3)
|
• Relocate “nitroglycerin protocol” icon and include descriptive label
• Increased clarity in displaying abnormal values
• Increased clarity in identifying graph icons
• Add “side effects” row in chart
• Display nitro DSS performance information in blood pressure graph window
• Enhance graph readability by displaying numeric values
• Include map with predicted blood pressure
|
• Move the “nitroglycerin protocol” icon to the top right corner and change from Nitroglycerin Protocol to Link to Nitroglycerin
• Remove ! for abnormal values
• Instead of word “Actual” use graph icon. Also position this icon in the same area
as blood pressure (MAP)
• Create another row in the chart that is “side effects” In this, it is a free text
option for nurses to add info
• In the blood pressure graph window included the nitro DSS' performance on patients
from that hospital. Included, “Overall absolute error is 5 mm Hg” and an information icon that when clicked it states, “This is the error rate (difference between actual and predicted blood pressure) for
the system when tested on 1,000 THP patients.”
• In the blood pressure graph, include the numeric value for actual and predicted
blood pressure when hovered over the different time intervals.
• Include mean arterial pressure into the prediction so it appears with the blood
pressure
**Incorporated a scenario to have the dose change from 15 μg/min with a blood pressure
of 134/77 mm Hg to 20 μg/min with a predicted blood pressure of 133/73 mm Hg for the
first prediction. For the third prediction the blood pressure was 90/60 mm Hg with
a dose of 30 μg/min and increased to 175/88 with a dose change to 20 μg/min
|
|
Round 3 (version 4)
|
• Clarify nitroglycerin protocol label
• Streamline side effects options
• Include nitroglycerin start date/time
• Change blood pressure graph icon color
• Option for predicted blood pressure display
• Revise “cancel dose” to “reject” with free text
|
• Make the Link to Nitroglycerin Protocol appear like a hyperlink (blue + underline)
• Make the side effects a drop-down option
• Diaphoresis
• Dizziness
• Headache
• Vision changes
• Other_____________
• Include Nitroglycerin Start Date/Time: August 6, 2023, @1100 under the medication
name
• Change blood pressure graph icon to color blue
• Remove “X” on the alert pop-up as it has same purpose as okay option
• Created an option to show predicted blood pressure below the actual blood pressure
in the table. Users can select if they want to view predicted blood pressure and unselect
if they do not.
• Changed the cancel dose option to reject and nurses can free text why they are
rejecting the potential new dose/predicted blood pressure
|
Abbreviation: EHR, electronic health record; nitro DSS, Nitroglycerin Dose Titration Decision Support System; RR, respirarion rate.
Usability
All the participants' ratings on SUS exceeded the threshold score of 68. Overall,
the usability score decreased slightly after the second round of usability testing
(from mean of 90.5 to 86). Additional features were incorporated and tested in round
2 based on the feedback from round 1. However, some nurses shared that these changes
made the prototype more complex. For instance, nurses found the Nitroglycerin Protocol icon ambiguous and suggested it be better identified and separated from the accept
and cancel buttons (see [Fig. 3]). This feature was revised to include a descriptive label, Link to Nitroglycerin Protocol, placed at the top right corner, and the reaction from the nurses was favorable to
this modification (see [Fig. 3]). The nurses also reported the need to incorporate a function to record patients'
side effects. Initially, this was designed as a free-text feature. However, nurses
desired a drop-down menu to select side effects rather than manually entering the
side effects (see [Fig. 3]). These findings in the usability testing aligned with the theme of a clear and consistent interface as identified in the qualitative study phase.
Another feature from round 2 that nurses wanted modified to improve usability was
the presentation of abnormal values. The prototype in round 2 identified all predictions
of blood pressure that fell outside normal limits in red color and with an exclamation
mark (i.e., !90/60). The nurses found this change misleading as the exclamation mark
can be mistaken for the number 1 reading, a blood pressure of 190/60. Abnormal values
were revised to remove the exclamation mark (i.e., 90/60; see [Fig. 3]). Furthermore, in round 2, the word Actual was used to indicate the presence of a graph that would be accessible to nurses when
they clicked on this feature. However, using the word, Actual, confused the nurses as this term does not inherently symbolize or imply the presence
of a graph in a typical user interface. Nurses suggested changing Actual to a graph icon for clarity (see [Fig. 3]).
Another potential contributing factor to the reduced usability score in round 2 is
that a blood pressure prediction that the nurses may consider unlikely, given the
current blood pressure and potential change in dose, was introduced into the prototype.
The blood pressure was 134/77 mm Hg at 15 μg/min, and with a dose increase to 20 μg/min,
the predicted blood pressure was 175/88 mm Hg. The requirement to consider situations
where the decision support system provides a prediction that does not align with their
intuition yielded some valuable insights that enhanced the user interface. Nurses
suggested including the predicted mean arterial pressure to help assess whether to
accept or reject the dose. The nurses also requested a feature that identifies the
mean absolute error (difference between the actual and predicted blood pressure) related
to nitro DSS's performance within their institution to determine the practical use of the prototype
(see [Fig. 4]). These insights during the usability testing phase reflected the theme of reliability.
Overall, the inclusion of this scenario in round 2 prompted nurses to scrutinize the
utility of nitro DSS, which, in turn, fostered valuable input aligned with the theme of reliability, leading
to improved usability of nitro DSS.
Fig. 4 Displays the original and final version of the graphs for the nitro DSS prototypes. The original version (on the left) includes a graph of the actual and
predicted systolic blood pressure over time. The final version (on the right) includes
the actual and predicted systolic blood pressure, a graph of the nitroglycerin dose
over time, and the absolute error rate for nitro DSS when tested on patients from that organization. BP, blood pressure; nitro DSS, Nitroglycerin Dose Titration Decision Support System.
Acceptability
The acceptability scores for each participant and the mean score from each round of
usability testing are reported in [Fig. 2]. The scores of the OADRI showed a trend of increasing average acceptability, consistent
with participant responses during interactions with consecutive prototypes and comments
during the interview, indicating reduced barriers and enhanced facilitators related
to whether nitro DSS would be adopted.
All the nurses confirmed that they had not previously encountered a CDSS for titrating
nitroglycerin doses. It was their first exposure to a system that predicted a physiological
indicator for a titration task. Based on the items in the OARDI, all nurses moderately
or strongly agreed that patients would benefit from using the decision support system,
the system's wording is unambiguous, and it would be useful for their practice.
For nurses, a key determinant of accepting nitro DSS was ensuring a low error rate between the actual and predicted values. During the
debrief interview, nurses said there is little value in adopting nitro DSS in their practice if it is inaccurate. Nurses wanted evidence about the accuracy
of the predictions within their institution and patient population rather than academic
papers. Nurses conveyed that knowing nitro DSS's performance within their institution is crucial, as it assists in making an informed
decision about using it in their clinical practice.
Nurses also expressed the importance of a feedback mechanism to reduce further the
error rate between actual and predicted blood pressure in order to accept the CDSS.
A feedback mechanism for continual improvement of the prediction component allows
users to comment on the prediction, making it easier to gather user input to improve
the performance of nitro DSS. A new feature that authorized nurses to reject the prediction, accept the dose,
and share their reason for rejecting it was added (see [Fig. 5]). Nurses wanted this feature to help train the algorithm and improve the overall
performance of nitro DSS. This design modification reflected the theme of reliability, and nurses appreciated
having this feature to support the acceptability of nitro DSS.
Fig. 5 Displays the feature to report the reason for rejecting the prediction (final prototype;
data are imaginary).
Nurses expressed having minimal data entry requirements that would encourage them
to adopt nitro DSS. We translated this need by including three data entry features: potential new dose,
patient side effects, and reason for rejection. As mentioned previously, patient side
effects were revised to have a select all that apply feature, and reason for rejection
is not required for each output.
Having aesthetically appealing alerts to catch attention was mentioned as a determinant
of acceptability during the debriefing interview. In usability testing round 1, the
nurses shared that they disliked the critically low blood pressure alert on the right
side of the screen, as it was not eye-catching. The prototype was modified to place
the critically abnormal blood pressure alert in the center of the screen (see [Fig. 6]). Alluding to the theme of a vigilant interface from the interviews, this change
was sustained in all subsequent rounds of usability testing, and nurses shared positive
feedback and appreciated the revision.
Fig. 6 Displays the alert for abnormal blood pressure prediction on the original (top) and
final (bottom) prototypes (data are imaginary).
Discussion
Our study findings highlighted interoperability as an essential aspect of designing
a CDSS. Nurses expressed the need for nitro DSS to communicate with clinical information systems, specifically EHR, and exchange
information seamlessly. Consistent with prior research, interoperability features
are integral to developing and implementing CDSSs.[40]
[41] A qualitative study explored the challenges in developing, managing, and using CDSSs,
and interoperability was identified as a requirement by the nurses to reduce the burden
of manual data entry and prevent double documentation.[42] Minimizing data entry requirements decreases the time spent using the system, allocating
more time to direct patient care.[43] Furthermore, clinicians and CDSS vendors are frustrated with the minimal progress
made to have CDSS interoperable with EHRs[42]: These findings add to the present study the importance of collaboration between
CDSS developers and EHR vendors to ensure interoperability. Double documentation burden
could be reduced if nitro DSS and existing systems exchange information; for example, patients' vital signs can
be imported from the EHR, and dosages can be translated into the MAR, rather than
nurses manually inputting this information. Reducing double documentation minimizes
repetitive tasks that are time-consuming and prone to error and can lead to inconsistencies
or discrepancies in data, ultimately creating inefficiencies.[43]
Ensuring interoperability among systems could enhance the accuracy of nursing documentation.
Tung et al (2022) evaluated timestamp discrepancies between EHR documentation and
pump event logs (actual administration process) for vancomycin infusion.[44] The researchers used process mining to examine the conformance between pump event
logs and EHR data for a single hospital in the United States.[44] An algorithm was developed to link records of the same infusions, and discrepancies
in infusion start time, completion time, and interruptions were analyzed. Of the 1,858
infusions, approximately 20% had an infusion start time discrepancy of more than ±10 minutes.
The researchers suggested these discrepancies are inherent parts of the current EHR
documentation process commonly found in hospitals, not unique to the study site.[44] A potential solution to reducing the discrepancy in documentation can be to have
interoperable systems. Suppose a dose accepted on nitro DSS is automatically documented in the MAR section of the EHR; this could eliminate the
need to document it in the EHR and potentially reduce the time discrepancies.
Nurses possess a wealth of knowledge and expertise that can be leveraged to create
more tailored and effective CDSS.[45]
[46] Since nurses will be the end-users of nitro DSS, their involvement in the development process could significantly shape the acceptance
and use of nitro DSS in clinical practice. In our study, nurses identified essential design specifications
of nitro DSS. Nurses shared that if the interface is centered on their needs, it will be easier
to use and more prone to adoption in their clinical practice. Incorporating user-identified
preferences into user interface design aligns with the insights from the technology
acceptance model (TAM), a well-recognized conceptual model used for information systems
research.[47] TAM is used to understand why users accept or reject information technology.[48] TAM suggests that individuals' beliefs about the usefulness and ease of using technology
have led users to (1) form positive attitudes toward technology, (2) develop intentions
to use that technology, and (3) actually use the technology.[47]
[48] Not having the end users involved in artificial intelligence technology development
can lead to interfaces that inadequately capture users' requirements, potentially
resulting in low adoption rates.[49] SUS is often used to assess a system's perceived ease of use, and this study's results
indicate excellent usability of nitro DSS.[50] According to the TAM, perceived ease of use is a precursor to technology acceptance.[51] Therefore, the results denote a favorable foundation for the technology's eventual
use.
Nurses' involvement in usability testing research can provide an opportunity to understand
the prototype's functionality and purpose, potentially reducing emotions of fear,
doubt, and frustration tied to using decision support systems in practice.[52] Previous work highlighted nurses' resistance to adopting CDSSs in their practice,
demonstrating that they would repeatedly double-check the reliability and credibility
of the CDSS because they doubted the algorithm's accuracy.[52] In our study, the reliability of the CDSS was also a theme identified from the qualitative
study findings and an essential component of usability and acceptability. Their involvement
allowed them to share preferences of design specifications that can be incorporated
into nitro DSS to develop trust. A critical factor in adoption is trust, which can be bolstered
through transparent information on underlying evidence, collaboration with end-users,
and feedback about users to drive continuous improvement.[53] Nurses wanted to know that blood pressure predictions within their organizations
were accurate, as this can help them to make an informed decision on whether to accept
or reject those predictions. Clinicians' trust in predictive CDSSs is influenced by
their perceptions of the accuracy and correctness of the CDSSs, and the more accurate
the prediction, the greater the trust in the CDSS.[55] For the nitro DSS, a graph showing the actual and predicted blood pressure over time and mean absolute
error (difference between the actual and predicted blood pressure) related to the
nitro DSS model performance was embedded into the interface. Furthermore, the involvement of
clinicians in developing the CDSS is an essential factor contributing to trust. Clinicians'
ability to understand the system (perceived understandability) is also an essential
factor influencing trust.[54] CDSS developers need to involve end-users in developing CDSS prototypes and integrating
features that demonstrate the system's reliability to gain clinicians' trust.
Limitations
The study sample included a relatively small number of nurses from two separate hospitals
of a single organization in Canada. Nurses with various years of work experience helped
design the user interface, but only nurses who worked in the coronary care department
were included. Nurses in other departments, such as the emergency department, use
nitroglycerin for angina relief, so their preferences for nitro DSS may differ. Furthermore, nurses in this study highlighted the importance of aligning
the nitro DSS interface with their existing EHR, recognizing other organizations may not use the
same EHR system. Readers should reflect on these factors when considering the transferability
of the findings to other contexts.
The interviewer was a research team member who designed the initial prototype based
on results of phase one. It is possible that nurses were hesitant to share opinions
criticizing the system. The interviewer reassured the nurses that they could openly
share their feedback on the system as it would help to improve the design.
In this study, three rounds of usability testing were conducted to develop a nitro DSS prototype. The subsequent research should include developing a high-fidelity prototype,
testing in silent trials, and conducting a randomized control trial. Additional research
with a larger, more diverse sample of nurses from multiple organizations across Canada,
employed in different clinical settings, and using various EHRs is needed to develop
a high-fidelity nitro DSS prototype and test usability and acceptability further. Once a high-fidelity prototype
is developed, the accuracy of blood pressure predictions generated by nitro DSS in real-time can be evaluated using silent trials. The benefit of silent trials is
that models are evaluated on prospective patients in real time while the end-users
are blinded to the model output (blood pressure predictions), meaning their clinical
decision-making is unaffected.[55] The next step would be to conduct a randomized controlled trial in the coronary
care units to assess the accuracy and use of nitro DSS. The measures evaluated would include the number of hypotension and hypertensive
episodes, the number of titrations required to achieve relief of angina, and nurses'
nitro DSS usage patterns (i.e., number of missed alerts or delays as measured in minutes in
responding to alerts, usability, and acceptability).
Conclusion
Through a UCD approach, this study developed a user interface prototype that interacts
with nurses to titrate intravenous nitroglycerin dosage. This study used an in-depth
qualitative phase to pinpoint design specifications to inform a prototype that later
underwent three iterative usability testing rounds to enhance the user interface.
The refined nitro DSS prototype incorporates features like side effect reporting, easy access to a summary
measure of the blood pressure prediction accuracy for the local context, a reporting
mechanism to record if a user considered the blood pressure prediction was unlikely
to be correct, and minimal alerts to prevent the burden of alert fatigue. The study
highlighted that nurses found the final nitro DSS prototype to have excellent usability and acceptability. Further research is required
to evaluate the usability and acceptability of nitro DSS in direct patient care. The involvement of nurses in user interface development is
crucial, as demonstrated by this study, which showed that their input led to a usable
and acceptable prototype.
Clinical Relevance Statement
Clinical Relevance Statement
Nurses' involvement in interface development is essential to generate design specifications
that enhance the usability and acceptability of CDSSs. The final iteration of the
nitro DSS prototype incorporates nurses' needs for a reliable, interoperable, vigilant system
with a clear and consistent interface. This study can guide designing the interface
of CDSSs for titrating other vasoactive agents.
Multiple-Choice Questions
Multiple-Choice Questions
-
What is a think-aloud technique?
-
Users express their thoughts, feelings, and opinions while interacting with the prototype.
-
Observing individuals performing their typical clinical workflows.
-
Patient charts are gathered to answer a clinical question.
-
Participants complete a survey.
Correct Answer: The correct answer is a. The think-aloud technique involves participants talking
aloud while interacting with the prototype and expressing their user interface perspective.
-
What is the definition of a clear and consistent interface as defined in this study?
-
Actively monitors patient predictions, prioritizes alerts based on deviations from
normal ranges, and aims to enhance patient safety without burdening nurses with excessive
or unnecessary alerts, thereby addressing the issue of alert fatigue in health care.
-
Design and present the interface in an easily understandable and uniform way with
existing interfaces, creating an intuitive experience.
-
Exchange data seamlessly and efficiently, interact with other systems to work together,
and share information to enable efficiency.
-
Consistently generating accurate predictions, demonstrating a high level of trustworthiness
in its predictive capabilities, and having transparency in its performance
Correct Answer: The correct answer is b. A clear and consistent interface refers to the interface's
design and presentation of the interface, and it is designed in a way that makes it
easily understandable and uniform with existing interfaces, thereby creating an overall
intuitive experience.
