Keywords
clinical decision support - user testing - antimicrobial stewardship
Background and Significance
Background and Significance
Antimicrobial resistance is a major public health threat. Each year, antibiotic-resistant
bacteria account for 2.8 million infections and 35,000 deaths in the United States.[1] A large contributor to the development of antibiotic resistance is poor adherence
to guideline-recommended antibiotic prescribing. In the emergency department (ED),
approximately 30 to 50% of the 10 million outpatient antibiotic prescriptions are
inappropriate or unnecessary.[2]
[3] Consequently, multiple national organizations have emphasized the need for antimicrobial
stewardship programs (ASPs) in the ED.[4]
[5]
[6] ASPs are effective for inpatient settings, reducing costs and the impact of resistant
bacteria through the promotion of narrow-spectrum and guideline-adherent antibiotic
prescribing.[7]
[8]
[9]
[10] However, there are barriers to ASP implementation in the ED, including erratic workflow,
shift work by clinical providers, and the need for empiric therapy in the absence
of confirmed diagnosis.[11] In addition, the failure to adapt national guidelines into ED local context has
impaired uptake of antibiotic-prescribing guidelines.[12]
Our investigative team recently determined that automated clinical decision support
(CDS) systems are strongly preferred by ED and hospital antimicrobial stewardship
leaders.[13] Using a user-centered design approach, our team rigorously developed a prototype
electronic health record (EHR)-based CDS to improve guideline-adherent prescribing
for two high-priority pediatric infections, community-acquired pneumonia (CAP), and
urinary tract infections (UTIs).[14]
[15] The prototype antimicrobial stewardship CDS presents locally adapted prescribing
recommendations and accounts for ED provider preferences and workflow.[15] The long-term goal is to create a platform-agnostic CDS that can be used interchangeably
within EHRs and be widely implemented in EDs.
User-centered methods in the evaluation, development, and deployment of EHRs, and
specifically CDS, are recommended to improve system usability. Formal usability testing
is essential to evaluate the effectiveness of health information technology,[16] though underused even in initial development and configuration of the hospital-based
EHR.[17]
[18] Development of effective CDS with user-centered design methods has the potential
to reduce cognitive burden, improve efficiency, and minimize EHR burnout in providers.[19]
[20]
[21]
[22] Furthermore, CDS has the potential to drive clinical improvements including adherence
to clinical guidelines and evidence-based care for common infectious diseases such
as CAP.[23]
[24]
[25] Using a simulated ED environment, the objective of our pilot study was to test the
usability and effectiveness of a new, prototype CDS developed through user-centered
design for antibiotic prescribing for pediatric CAP and UTI.
Methods
Study Design
This was a user-testing study comparing a CDS-enhanced EHR order set (prototype CDS)
for outpatient antibiotic prescribing for pediatric CAP and UTI, according to the
current standard EHR order set used in our ED. The study was deemed exempt by our
institutional review board.
Study Setting and Participants
This was a scenario-based laboratory study conducted among ED providers from a large,
tertiary care, academic pediatric health system participating in the Pediatric Emergency
Care Applied Research Network (PECARN). The EDs within the health system consist of
a tertiary care referral ED, three community-based EDs, and two free-standing urgent
care (UC) centers that receive approximately 170,000 visits annually. All sites operate
on the same EHR (Epic Systems Inc., Verona, Wisconsin, United States); the appearance
and function of the EHR are identical at all sites. The ED provider group consists
of over 150 pediatric emergency medicine specialists, emergency general pediatricians,
and advanced practice providers and trainees. For the study, we purposively sampled
ED providers using e-mail invitations and targeted a variety of providers that clinically
practice in the EDs and UCs. Verbal consent was obtained at the start of user testing
sessions.
Description of the Prototype Clinical Decision Support for Community-Acquired Pneumonia
and Urinary Tract Infection Antibiotic Prescribing
Using user-centered design principles, our study team developed a prototype CDS for
ED antibiotic prescribing with input from three PECARN institutions to enhance the
generalizability of the CDS content and activation. Our team formally adapted CAP
and UTI CDS antibiotic recommendations into the local context to account for current
institutional pathway recommendations, antibiotic resistance patterns, and pharmacy
availability.[26]
[27] We refined guideline-adherent recommendations using patient characteristics, such
as age, presence of fever, penicillin allergy, and results of diagnostic testing;
the team translated these adapted prescribing recommendations into algorithms which
were included in the CDS prototype. Key stakeholder interviews and provider focus
groups informed CDS format and strategies for integration into ED care, formal workflow
observation and analysis informed EHR triggering and presentation of the CDS.[14]
[15] Our overall approach used passive CDS techniques rather than hard stops. The CDS
was designed in accordance with the five rights of CDS and underwent heuristic review
by investigators and users to ensure functionality.[28] The prototype CDS consisted of orders embedded in the usual patient discharge workflow
to guide end-users toward the correct antibiotic choice, paired with best practice
advisories (BPAs) displays of informational resources regarding CAP and UTI antibiotic
prescribing ([Fig. 1]). The CDS also included a notification banner alerting providers if a patient had
a prior diagnosis of a UTI and directing users to review prior cultures/sensitivities.
Fig. 1 Explanatory text for CAP and UTI present in prototype CDS. CAP, community-acquired
pneumonia; CDS, clinical decision support; PCN, penicillin; UTI, urinary tract infection;
Study Procedure and Usability Testing Sessions
We used a scenario-based approach to compare the prototype CDS to the current approach
for antibiotic prescribing in the EHR.[29]
[30] A simulated EHR test environment was created with realistic clinical scenarios for
CAP and UTI ([Supplementary Appendix], available in the online version). In total, there were six simulated patients for
CAP and eight for UTI. Simulated patients were designed to address and represent various
factors present in the aforementioned antibiotic-prescribing algorithms. Each case
scenario could be used to measure the current approach to antibiotic prescribing or
to test the usability of the prototype CDS.
We conducted usability testing with three simulated patients for each participant.
The first case presented to the participant tested the current ordering approach,
followed by two cases using the prototype CDS. Simulated patients were selected at
random for each user test to ensure that the cases were distributed among the participants
in a balanced fashion.[31] The first case assigned to participants was randomized to be either a CAP or UTI
patient, and the second and third cases included both a CAP and UTI patients presented
in random order.
Participants were instructed to read each virtual case as presented on paper, and
then review the patient chart in the EHR as desired, with the intent of ultimately
discharging the patient from the ED. Each participant completed the task of discharging
their first assigned patient using the current approach to outpatient prescribing
in the existing (unenhanced) EHR workflow. The next two simulated patients presented
were designed to trigger the appearance of the prototype CDS in the EHR discharge
workflow. The appearance of the CDS was specifically triggered by the entry of one
of a preselected set of discharge diagnoses for CAP or UTI. For the discharge diagnosis
to activate the CDS, there had to be a recent order for a chest X-ray for patients
presenting with CAP or a positive urinalysis result for patients who presented with
symptoms of a UTI. These orders and results were built into the simulated patient
scenarios prior to user testing in order to direct the end-user toward discharge diagnoses
and antibiotic prescribing rather than diagnostic workup. Participants independently
conducted all antibiotic ordering tasks; redirection was briefly provided by the study
team observer only if necessary. Specific participant behaviors were documented on
a standardized data collection form. Testing sessions were conducted by one of two
observers (E.A.M. and R.D.M.), who instructed participants to follow “think-aloud”
protocols throughout each case scenario, which called for participants to verbalize
all thoughts as they interact with the EHR.[32] In addition to recording objective measures, observers took qualitative notes as
participants made think-aloud statements. Each user testing session took approximately
25 to 30 minutes.
Study Phases
Our study was conducted in two phases. The intent of having two phases was to determine
if major functional issues existed with the prototype CDS, or if minor adjustments
were needed to improve appearance or usability. Phase 1 involved initial user testing.
After nine user-test sessions in Phase 1, it was determined that only minor design
and workflow changes were recommended for the prototype CDS, based on participant
feedback. Subsequently, 12 user tests were conducted phase after prototype CDS revision.
We present the majority of results as overall (Phase 1 and Phase 2 combined) as iterative
changes were expected to be minor and not expected to substantially change our results.
However, we analyzed and present our provider assessments by each phase of the study,
as well.
Outcome Measures of User Testing
The study team identified seven functions of the CDS to assess provider satisfaction
with current EHR ordering and the prototype CDS approach to antibiotic prescribing
([Table 1]). Each item was rated on a five-point Likert scale (1—very dissatisfied; 5—very
satisfied). After participants worked through each clinical scenario, they completed
assessments specific to the CAP and UTI cases and each diagnosis-specific CDS. Provider
perceptions of the usability of the current and CDS enhanced order set were also assessed
through participant surveys; domains assessed in these are summarized in [Table 2]. We utilized five-point Likert scales to assess agreement with the domains of the
CDS (1—strongly disagree; 5—strongly agree).
Table 1
Domains assessed for provider satisfaction with the prototype CDS and disease-specific
scenarios using five-point Likert scales
|
Domain
|
Assessment description
|
|
Antibiotic prescribing well structured
|
Assess whether the disease-specific CDS is well structured and well-integrated into
the ED EHR workflow
|
|
System helped with patient management
|
Assess whether the disease-specific CDS improves outpatient antibiotic prescription
and overall patient management in the ED EHR workflow
|
|
Preferred over prior methods
|
Assess whether the user prefers the disease-specific CDS over the current EHR workflow
|
|
Saved time
|
Assess whether the prototype CDS saves time compared with the current EHR workflow
for the disease-specific antibiotic prescription process
|
|
Overall satisfaction
|
Assess overall satisfaction with how the CAP or UTI CDS is designed to aid providers
in outpatient antibiotic prescribing
|
Abbreviations: CAP, community-acquired pneumonia; CDS, clinical decision support;
ED, emergency department; EHR, electronic health record; UTI, urinary tract infection.
Table 2
Functions of the CDS used to assess provider perceptions of usability
|
Function
|
Function description
|
|
Patient identification
|
Identification of patients with evidence of CAP or UTI
|
|
Antibiotic selection
|
Ease of selecting appropriate antibiotics for treatment of CAP or UTI
|
|
Integration into workflow
|
Ease of outpatient antibiotic prescription in the ED EHR workflow
|
|
Suggestion of antibiotic alternatives
|
Provision of specific and appropriate alternative antibiotics when patients have antibiotic
allergies
|
|
Explanation of suggested antibiotics
|
Provision of explanations for recommended antibiotic choices
|
|
Provision of prescribing resources
|
Provision of additional resources to help the user better understand suggested antibiotic
choices
|
|
Overall satisfaction
|
Overall satisfaction with how the CDS is designed to aid providers in outpatient antibiotic
prescribing
|
Abbreviations: CAP, community-acquired pneumonia; CDS, clinical decision support;
ED, emergency department; EHR, electronic health record; UTI, urinary tract infection.
The study assessed the ordering systems using established usability testing and performance
measures, including task completion and errors related to decision-making and system
usability.[32] Decision-making outcomes included participant choices for appropriate CAP and UTI
treatment actions, including the selection of the appropriate diagnosis to trigger
the CDS order set and the choice of guideline-adherent antibiotic therapy for CAP
and UTI. Usability outcomes included those related to appropriate participant actions
used to engage with the ordering system, including interacting with best-practice
alerts and completion of all prescribing tasks. For the purposes of the study, we
calculated rates of errors in participant decision-making and CDS usability. “Decision-making
errors” were defined as errors in clinical decisions and included failure to activate
the CDS with an appropriate diagnosis and prescription errors: ordering a nonguideline-adherent
antibiotic, not ordering an antibiotic when one was warranted or ordering an antibiotic
when not recommended per local antibiotic-prescribing guidelines. “Usability errors”
were related to system usability or errors made causing deviations from the most direct
and correct pathway to patient discharge. Usability errors included failure to select
the correct discharge order set without prompting or prescription of the incorrect
antibiotic only if the user subsequently corrected the order based on CDS recommendations
(correcting an otherwise decision-making error). The occurrence of decision-making
and usability errors was compared between current prescribing workflow and the prototype
CDS.
The study measured workload using the National Aeronautics and Space Administration
Task Load Index (NASA-TLX).[33]
[34]
[35] The NASA-TLX is a multidimensional scale designed to obtain workload estimates from
one or more operators, while they are performing a task or immediately afterward and
has been used to evaluate EHR-based tools.[36]
[37] Study observers administered the NASA-TLX to participants twice: once after the
completion of the current antibiotic-prescribing method test case, and a second time
after the completion of both scenarios using the prototype CDS. Participants completed
a posttest survey after finishing all cases and NASA-TLX tools.
Data Collection and Analysis
The study team summarized all quantitative data using standard descriptive statistics.
Planned comparisons included those between the current ordering process and the prototype
CDS approach for all items, as well as between the two study phases to determine if
iterative changes were influencing testing results. Proportions were compared using
chi-square testing or Fisher's Exact test, where appropriate. For Likert scale items
and NASA-TLX, we calculated both medians and interquartile ranges (IQRs). Though most
data approximated normality, we used nonparametric statistics for comparison owing
to the limited sample size. Therefore, we also reported means and standard deviations
(SDs) for clinical relevance. Statistical comparisons were made using the Wilcoxon
signed-rank test as these represent a pre–post-analysis within the same population.
Analysis of the NASA-TLX raw scores was conducted as instructed by the developers,
including paired tests using the Wilcoxon signed-rank test.[38] Results of hypothesis testing were reported with both the effect size r = Z / √N, where Z is Wilcoxon's test statistic and N is the sample size, as well as p-values, with a significance level of <0.05.[39] Magnitude of effect sizes were interpreted according to Cohen's criteria where r = 0.2 small, r = 0.5 medium, and r = 0.8 large.[40] All quantitative data were analyzed using IBM SPSS version 27.0 (IBM Co, Armonk,
New York, United States). Qualitative comments were thematically summarized for presentation.
Results
Demographics of Study Participants
The study team conducted a total of 21 user tests. Nine providers participated in
Phase 1 of the study, and 12 providers participated in Phase 2. All results presented
are overall analyses with both phases combined (n = 21), unless otherwise stated. Participants included residents (n = 10; 47.6%), fellows and attendings in pediatric emergency medicine (n = 9; 42.9%), pediatricians (n = 1; 4.8%), and advanced practice providers (n = 1; 4.8%). Most participants worked as providers in the tertiary care referral ED
(85.7%). The median number of years in the practice was 4.5 years (range of 1.5–20
years). Most providers (81.0%) indicated that they prescribed outpatient antibiotics
on a weekly basis.
Overall Assessment of Current System and Prototype Clinical Decision Support
The prototype CDS exhibited moderate-to-large preferences in usability as compared
to the current EHR order set in each domain ([Table 3]). Overall results demonstrated that the prototype CDS exhibited significant improvement
in the identification of patients needing antibiotic treatment, antibiotic selection,
suggestion of antibiotic alternatives, and overall satisfaction ([Table 3]), with a large effect on the explanation of suggested antibiotics (r = 0.773) and the provision of prescribing resources (r = 0.848). These findings were also reflected in the qualitative data provided by
study participants. One participant commented “It's nice that the [link to the] clinical
pathway is right here,”, while another noted that they “really like the explanations
here [in the Best Practice Advisory box].” Compiled qualitative data are available
in [Supplementary Table S1] ([Supplementary Material], available in the online version).
Table 3
Perceptions of system usability assessed by participants (n = 21) for current and CDS ordering systems (5-point Likert scale: 1 = very dissatisfied;
5 = very satisfied)
|
Function
|
Current system
|
Prototype CDS
|
Effect size (r)
|
p-Value[a]
|
|
Median (IQR)
|
Mean ± SD
|
Median (IQR)
|
Mean ± SD
|
|
|
|
Patient identification
|
4 (3–5)
|
3.4 ± 0.8
|
4 (3–5)
|
4.2 ± 0.6
|
0.660
|
0.002
|
|
Antibiotic selection
|
4 (3.5–4.5)
|
3.8 ± 0.6
|
5 (4–5)
|
4.5 ± 0.8
|
0.612
|
0.005
|
|
Integration into workflow
|
4 (3–5)
|
3.8 ± 0.7
|
4 (3–5)
|
4.2 ± 0.8
|
0.42
|
0.053
|
|
Suggestion of antibiotic alternatives
|
3 (2–4)
|
3.2 ± 0.8
|
5 (4–5)
|
4.4 ± 0.8
|
0.62
|
0.004
|
|
Explanation of suggested antibiotics
|
3 (2–4)
|
2.6 ± 0.8
|
4 (3–5
|
4.0 ± 1.1
|
0.773
|
<0.001
|
|
Provision of prescribing resources
|
3 (2–4)
|
2.5 ± 0.6
|
5 (4–5)
|
4.4 ± 0.8
|
0.848
|
<0.001
|
|
Overall satisfaction
|
4 (3–5)
|
3.5 ± 0.6
|
5 (4–5)
|
4.4 ± 0.8
|
0.700
|
0.001
|
Abbreviations: CDS, clinical decision support, IQR, interquartile range, SD, standard
deviation.
a Statistical comparisons made using the Wilcoxon Signed-Rank test. Effect size reported
using effect size r = Z/√n, where Z is the Wilcoxon test statistic and n is the sample size.
Assessment of Current System and Prototype Clinical Decision Support: Phase 1 to Phase
2
Qualitative comments from Phase 1 participants suggested minor design and workflow
changes were warranted. Trends toward preferences for the prototype CDS were present
in Phase 1 of the study, though only the provision of prescribing resources reached
statistical significance ([Table 4]). Phase 1 participants provided feedback regarding the number of appropriate antibiotic
choices offered for the treatment of CAP in penicillin-allergic patients. In Phase
1, the CDS listed three cephalosporins as equivalent antibiotic choices for penicillin-allergic
patients. One participant commented, “There are too many [health system] sanctioned
cephalosporin options.” Another participant commented that it would be nice to stratify
these options by cost or to include a comment stating that they are all equivalent
if this were truly the case. Phase-specific data are present in the [Supplementary tables S2] and [S3] ([Supplementary Material], available in the online version).
Table 4
Function satisfaction by phase as measured by 5-point Likert scale[a]
|
Phase 1 (n = 9)
|
Phase 2 (n = 12)
|
|
Function
|
Current System
Median (IQR)
|
Prototype CDS
Median (IQR)
|
Effect Size (r)
|
p-Value
|
Current System
Median (IQR)
|
Prototype CDS
Median (IQR)
|
Effect Size (r)
|
p-Value
|
|
Patient identification
|
4 (3–5)
|
4 (3–5)
|
0.544
|
0.102
|
3.5 (2.5–4.5)
|
4 (3–5)
|
0.761
|
0.008
|
|
Antibiotic selection
|
4 (3–5)
|
4 (2–5)
|
0.360
|
0.279
|
4 (3.25–4.75)
|
5 (4.25–5)
|
0.833
|
0.004
|
|
Integration into workflow
|
4 (4–4)
|
4 (3–5)
|
0
|
1.000
|
4 (3–5)
|
5 (4–5)
|
0.709
|
0.014
|
|
Suggestion of antibiotic alternatives
|
3 (2–4)
|
4 (2–5)
|
0.393
|
0.238
|
3 (1.25–4.75)
|
5 (4–5)
|
0.763
|
0.008
|
|
Explanation of suggested antibiotic choices
|
3 (1.5–4.5)
|
4 (1.5–5)
|
0.602
|
0.071
|
2.5 (1.5–3.5)
|
4.5 (3.5–5)
|
0.879
|
0.002
|
|
Provision of prescribing resources
|
2 (1–3)
|
4 (2.5–5)
|
0.853
|
0.011
|
3 (2–4)
|
5 (4–5)
|
0.873
|
0.002
|
|
Overall satisfaction
|
4 (3–5)
|
4 (3–5)
|
0.471
|
0.157
|
4 (3–5)
|
5 (4.25–5)
|
0.833
|
0.004
|
Abbreviations: CDS, clinical decision support, IQR, interquartile range.
a Statistical comparisons made using the Wilcoxon's signed-rank test. Effect size reported
using effect size r = Z / √n, where Z is Wilcoxon's test statistic and n is the sample size.
In response to participant feedback in Phase 1, the study team made several refinements
to the prototype CDS. After discussion with institutional experts, we narrowed the
suggested antibiotic choice for penicillin-allergic children with CAP to a single
cephalosporin option. Of note, all three previous cephalosporin options were guideline
adherent; therefore, the limitation to a single option simplified but did not influence
the proportion of “correct” selections. To enhance CDS activation for CAP, trigger
criteria were expanded to include multiple variations of the chest X-ray order (1
view, 2 view, and 3 view). Finally, the explanatory text content in the informational
BPA was revised using feedback in order to be more streamlined in presentation. The
revisions also included the addition of information regarding antibiotic prescription
suggestions for penicillin-allergic patients. The results from Phase 2 of this trial
demonstrated statistically significant improvement in all ordering functions assessed
using the prototype CDS as compared to the current system, with high-moderate-to-large
effect sizes ([Table 4]).
Disease-Specific Results for Community-Acquired Pneumonia and Urinary Tract Infection
For pediatric CAP, the prototype CDS led to an improvement in participant selection
of the correct antibiotic for the treatment of CAP as compared with the current order
set (55 vs. 86%) ([Table 5]). There was a decrease in the number of decision-making errors made by participants
with the CDS; 64% of participants made decision-making errors when using the current
EHR order set and only 14.3% of participants made decision-making errors when using
the prototype CDS order set to prescribe antibiotics for the treatment of CAP ([Table 5]). Results from the pediatric UTI user tests were similar to those from CAP ([Table 5]). The implementation of the new prototype CDS led to a decrease in decision-making
errors made by participants, with 20% making decision-making errors with the current
EHR order set and only 9.5% of participants making decision-making errors when using
the new CDS. Usability errors were also less frequent with the new CDS (40 vs. 9.5%).
Provider interaction with associated informational BPA was greater for UTI than CAP.
Over half of participants either commented on or read the BPA content, though fewer
than one-third attempted to access local and national guidelines via embedded weblinks.
No participants interacted with or commented on the notification or banner indicating
that the patient had a history of a prior UTI diagnosis.
Table 5
Tasks assessed for CDS usability, decision-making errors, and usability errors in
simulated scenarios for CAP and UTI
|
CAP
|
UTI
|
|
Current order set (n = 11)
|
Prototype CDS (n = 21)
|
Current order set (n = 10)
|
Prototype CDS (n = 21)
|
|
Tasks
|
|
Decision-making outcomes
|
|
Diagnosis placed
|
10 (90.9%)
|
21 (100%)
|
10 (100%)
|
21 (100%)
|
|
Correct antibiotic Selected
|
6 (54.5%)
|
18 (85.7%)
|
9 (90.0%)
|
19 (90.5%)
|
|
Usability outcomes
|
|
Used order set
|
9 (81.8%)
|
21 (100%)
|
5 (50%)
|
21 (100%)
|
|
Signed order
|
10 (100%)
|
11 (100%)
|
10 (100%)
|
21 (100%)
|
|
Interacted with EHR elements
|
6 (54.5%)
|
12 (57.1%)
|
9 (90.0%)
|
18 (85.7%)
|
|
Commented on BPA
|
–
|
8 (38.1%)
|
–
|
12 (57.1%)
|
|
Content read in BPA
|
–
|
7 (33.1%)
|
–
|
9 (42.9%)
|
|
Clicked on BPA links
|
–
|
6 (28.6%)
|
–
|
6 (28.6%)
|
|
Completed prescribing tasks
|
11 (100%)
|
21 (100%)
|
10 (100%)
|
21 (100%)
|
|
Overall error rates
|
|
Decision-making errors
|
4 (63.6%)
|
3 (14.3%)
|
1 (10.0%)
|
2 (9.5%)
|
|
Usability errors
|
3 (27.3%)
|
3 (14.3%)
|
5 (50.0%)
|
2 (9.5%)
|
Abbreviations: BPA, best practice advisory; CAP, community-acquired pneumonia; CDS,
clinical decision support; EHR, electronic health record; UTI, urinary tract infection.
Provider satisfaction with the prototype CDS was high for both CAP and UTI ([Table 6]). Ratings were particularly high for UTI as compared to CAP, though the CDS for
both infections demonstrated preferences greater than four out of five for the majority
of domains, including ease of use and preference over the existing ordered system
of antibiotic prescribing.
Table 6
Provider satisfaction with the CAP and UTI prototype CDS an using five-point Likert
scales
|
Domain
|
CAP(n = 21)
|
UTI(n = 21)
|
|
Median (IQR)
|
Mean ± SD
|
Median (IQR)
|
Mean ± SD
|
|
Antibiotic prescribing well structured
|
4 (3–5)
|
4.00 ± 0.89
|
5 (3.5–5)
|
4.24 ± 0.52
|
|
System helped with patient management
|
4 (3–5)
|
4.24 ± 0.77
|
5 (4–5)
|
4.33 ± 0.94
|
|
Preferred over prior methods
|
4 (3–5)
|
4.14 ± 0.85
|
5 (4–5)
|
4.29 ± 0.90
|
|
Saved time
|
4 (3–5)
|
3.81 ± 1.12
|
5 (3–5)
|
4.10 ± 1.1
|
|
Overall satisfaction
|
4 (3–5)
|
4.24 ± 0.77
|
4 (3–4)
|
4.10 ± 0.94
|
Abbreviations: CAP, community-acquired pneumonia; CDS, clinical decision support;
IQR, interquartile range; SD, standard deviation; UTI, urinary tract infection.
National Aeronautics and Space Administration Task Load Index
The study team analyzed NASA-TLX results by comparing overall results, results by
phase, and results for CAP and UTI cases. The NASA-TLX index results demonstrated
a trend toward lower average workload reported by participants for the current system
(median: 26, IQR: 11–41; mean ± SD: 30.1 ± 15.1) versus the prototype CDS system (median:
25, IQR: 10.5–39.5; mean: 25.3 ± 1), though not statistically significant (p = 0.117) and with limited effect size (r = 0.342). NASA-TLX scores reflected a trend toward reduction in participant workload
in phase 2 of this study (median: 25, IQR: 0–52; mean: 31.5 ± 18.9 reduced to median
21, IQR: 2–40; mean 23.8 ± 13.3; p = 0.107) after improvements were made in the CDS, compared with Phase 1 (median:
31, IQR: 18–43; mean 28.4 ± 8.6) reduced to median 25, IQR: 13–37, mean 27.3 ± 7.6;
r = 0.104; p = 0.635).
For the CAP CDS, the mean NASA-TLX score reported by participants for the current
order set was median: 35, IQR: 17 to 53; mean = ± 16.5 compared with median 23, IQR:
9 to 37; mean: 25.4 ± 11.8 reported for CAP cases completed using the prototype CDS
which was significantly improved (r = 0.617; p = 0.041). The mean NASA-TLX score for UTI CDS when using the current order set was
median 22.5, IQR: 6.5–38.5; mean ± SD: 24.6 ± 11.8 compared with a mean NASA-TLX score
of median: 25.5, IQR: 8–43; 25.2 ± 10.9 reported for UTI cases completed using the
prototype CDS (r = 0.071; p = 0.80).
Qualitative Comments from User Tests
Participant comments reflected strong preferences toward the prototype CDS and specific
components of the prototype CDS ([Supplementary Table] [[Supplementary Material]], available in the online version). Participants' statements included “I like it!
Nice and clean,” and favored “fewer clicks” required for the prototype CDS to complete
an antibiotic prescription. Several participants commented that what they liked most
about the prototype CDS was the new BPA, which included the suggestion about when
antibiotics are specifically not recommended. When asked whether they would know how
to navigate the prototype CDS within the EHR workflow without additional training,
16 of the 17 respondents agreed that they would and all 17 indicated that they would
trust the CDS system to give accurate recommendations regarding antibiotic treatment
choices.
Discussion
Our findings demonstrate that our prototype CDS for outpatient ED antibiotic prescribing
was preferred by users and led to improved adherence to antimicrobial stewardship
guidelines in simulation. The CDS, which was created through a user-centered design
approach, demonstrated improvements in several aspects of usability and potential
effectiveness. Through our simulated patient approach for pediatric CAP and UTI, the
novel prototype CDS increased satisfaction and function for users while also reducing
rates of decision-making and functional usability errors. Therefore, we were able
to meet a critical goal of many ED providers, using health information technology
to support a common aspect of clinical care, antibiotic prescribing, through interaction
with the EHR. These results support the use of our prototype CDS for outpatient antibiotic-prescribing
ED and the future implementation of EHR-embedded CDS approaches to provide antimicrobial
stewardship.
Our study supports the potential of CDS to improve outpatient antibiotic-prescribing
ED. However, our findings also support the importance of user-centered design for
the creation of an acceptable and feasible CDS. We developed our CDS using contextual
considerations for the ED setting to integrate adapted prescribing recommendations
and workflow considerations, as well as end-user preferences to further refine the
CDS design. Our findings build upon existing studies demonstrating that usability
testing combined with user-centered methods, including iterative design, results in
more usable CDS that has the potential to contribute to improved outcomes including
adherence to clinical guidelines, patient safety, and reducing EHR burnout.[41]
[42]
[43]
[44]
Among the design features in our prototype CDS is passive activation, which does not
require clinicians to engage or “seek” the prescribing recommendations. When designing
our CDS, passive activation was important among numerous stakeholders and ED clinicians.[14]
[15] Automated triggering and CDS activation contributed to perceived improvements in
ease of use and resultant frequency of guideline-adherent antibiotic prescribing.
We believe that the passive approach to our CDS, as opposed to the current EHR approach
which requires active seeking of recommendations and antibiotic choices, substantially
improved usability. Furthermore, our CDS trended toward a perceived reduced workload,
without additional steps or forced user functions. Therefore, input from end-users
at the onset of design led to the development of a more effective, better accepted,
and more accessible decision-making tool and suggest that such an iterative approach
be employed in future CDS design.
We developed standard EHR functionality which supports the application of user-centered
methods by hospitals in the configuration of the EHR.[17]
[18] This includes the use of CDS tools such as discharge order sets, BPAs, weblinks
to local and national guidelines, and institutional antibiograms. These EHR and CDS
components were well received; perhaps even more critical to success was the placement
of these outpatient prescribing tools at discharge, which is often most logical in
the ED workflow and decision-making process for outpatient treatment of many infections.
By integrating the CDS into a discharge “order set,” we were able to naturally place
recommendations at the point of care while avoiding unnecessary presentation of recommendations
when not indicated. Furthermore, we were able to prepopulate the antibiotic prescriptions
such that the user only needed to sign the prescription, avoiding the possibility
of calculation errors in dosing and nonguideline adherent prescription durations of
treatment.
Implementation of CDS through user-centered design, and subsequent rigorous user-testing,
resulted in a prototype CDS with great potential for ability to improve ED clinical
care using the EHR, a ubiquitous health care tool. Our CDS addresses the end-user
desire to have a lower workload at the time of prescribing (e.g., not needing to seek
allergy information) and to minimize clicks within the EHR using automated prescribing.
For quality improvement approaches to improve patient outcomes, such as antimicrobial
stewardship, using a CDS in our method of design facilitates the management of data
from multiple sources and optimizes ED care through the reduction of prescribing variability.
As demonstrated in our quantitative assessments and user qualitative comments, a user-friendly
CDS can substantially improve the provider experience while simultaneously increasing
guideline-adherent care, with potential for application to other ED conditions.
Limitations
There are limitations to our findings. First, we conducted the user testing in a simulated
patient environment using cases typical of presentations of CAP and UTI in healthy
children; therefore, it is possible that the findings may not reflect the complexity
of care in the clinical ED. We did attempt to create realistic clinical, patient-based
scenarios, and the study appearance of the EHR was identical to that used in patient
care. Though we organized our test case-scenarios randomly, we conducted user testing
with our current system before the two cases testing our novel CDS. All participants
in the study had been previously exposed to the current system in their everyday practice,
and thus, the assumption was made that there would be less potential for learning
from this environment than there would be through exposure to the new features of
the CDS including general practice guidelines. This ordering of the systems allowed
us to minimize learning effects and test the current system without influencing antibiotic
prescription practices. Our study was a pilot study of 21 users, limiting the determination
of effect size and significance of some of our measurements. Nonetheless, we were
we able to use a repeated measures approach which provided important data demonstrating
strong effects and significance with respect to usability and user preferences. We
combined user test assessments of attending faculty and trainees, who may vary in
pediatric experience and training. Lastly, we conducted our user tests in single health
system, which may limit applicability to EDs that vary in EHR, clinical practice,
and antibiotic availability.
Conclusion
This study employed usability testing methodology to analyze the integration of CDS
for antibiotic prescription in the outpatient setting into the ED EHR. We found that
an enhanced, passively activated CDS with best practice alerts was strongly preferred
over usual methods of antibiotic prescribing and demonstrated the potential for a
lower cognitive burden for ED clinicians. Future steps will include the conduct of
CDS implementation trials across multiple settings to test effectiveness.
Clinical Relevance Statement
Clinical Relevance Statement
User-centered design and iterative user testing are important for the development
of an acceptable and effective EHR-based CDS in the ED. The design of passive CDS
using key stakeholder and end-user engagement has great potential to augment guideline-adherent
antibiotic prescribing for common outpatient ED infections.
Multiple-Choice Questions
Multiple-Choice Questions
-
Which of the following are useful methods of obtaining data to inform the design of
clinical decision support systems that are optimized for end-users?
-
NASA Task Load Index measurement
-
Electronic health record vendor input
-
Creation of best practice alert
-
Clinician focus groups
Correct Answer: The correct answer is option d. Input from end-users is essential
to design effective interventions using the electronic health record and clinical
decision support. To maximize the appearance, activation, and usable information from
the intervention, engagement of end-users through focus groups and/or semistructured
interviews can glean critical data necessary for successful design. Data from workflow
observations and analysis can be equally important to design. With respect to clinical
decision support, user-centered design can assist in the effective creation and adherence
to the “Five Rights of Clinical Decision Support,” delivering the right information,
to the right person, in the right intervention format, through the right channel,
at the right time in workflow.
-
Important benefits of user-centered design for clinical decision support may include
which of the following?
-
Decreased workload
-
Increased patient length of stay
-
Integration in multiple EHR vendors
-
Increased insurer reimbursement
Correct Answer: The correct answer is option a. Ultimately, user-centered design for
clinical decision support is used to help deliver improved patient care, including
quality measures such as timely care, shorter length of stay, and adherence to the
standard of care. However, by using input from end-users and key-stakeholder who may
be affected by the clinical decision support, this intervention can also be used to
meet the needs of the care providers. A more seamless clinical decision support intervention
embedded in the electronic health record can help deliver faster care and mitigate
frustrations and cognitive delays and reduce the cognitive burden introduced in navigating
the electronic health record system.
