Keywords
drug alerts - drug fatigue - desensitization - salience rate - clinical decision support
system
Background and Significance
Background and Significance
The use of electronic health records (EHRs) has substantially transformed the practice
of medicine for all disciplines and subspecialties. The digitalization of medical
records has enabled timely medical documentation and has access to large volumes of
patient health information. Healthcare providers can now place orders, consult, and
communicate treatment plans with other specialists from local or remote workstations.
The U.S. government has incentivized the adoption of health information technology
(HIT), with legislations, such as the Health Information Technology for Economic and
Clinical Health (HITECH, 2009), which have further augmented the use of HIT.[1]
[2]
[3] Currently, more than 70% of healthcare systems in the United States have implemented
EHRs among their institutions.[4]
[5]
The increasing adoption of HIT has preceded the development of clinical decision support
(CDS) systems which encompass an array of software tools aiming to improve physician
performance.[6] CDS systems, now largely available across healthcare institutions, incorporate various
features such as documentation templates, clinical guidelines, diagnostic support,
drug alerts, and reminders.[7]
[8]
[9]
[10] The utilization of CDS has demonstrated different advantages for improving practitioner
performance and clinical care, including prevention of harmful prescriptions[11] and reducing adverse drug events.[8]
[10] These advantages, however, do not come without associated risks of legal liability
pointed out by physician making clinical diagnosis,[12] risks of alert fatigue, and negative impact on user skills by reliance on alert
system, as well as limitation of identifying appropriate alerts.[13]
[14] Drug alerts are triggered when a medical order is incompatible with a set of parameters
which may include dose ranges, frequency of administration, associated allergies,
and medical conditions, among others.[15] Healthcare providers may respond to drug alerts either by overriding or acknowledging
the alert via modification or cancellation of an order. Drug alerts, for example,
can be inappropriate, or may hamper workflow when excessive. Alerts may not account
for the complete clinical context in which a medical order is placed in the computerized
physician order entry system (CPOE).[16]
CDS acceptance is defined as compliance with the displayed CDS recommendations. Salience
is the percentage of alerts that a provider acknowledged by either a change or removal
of the order[17] and is often used as a primary outcome measure to estimate appropriateness of the
CDS systems. Prior studies have shown suboptimal salience rates among large healthcare
institutions, with average override rates of 46.2 to 96.2%.[18]
[19]
[20] Several authors have proposed different causes for lower salience rate such as “drug-alert
fatigue”[4]
[21]
[22] caused by excessive or inappropriate drug alerts, or “desensitization,”[23] secondary to prolonged or repetitive exposure to alerts, with some suggesting that
these may lead to an increased risk for medical events.
Salience rates vary between different providers based on multiple factors, such as
degree of medical training, experience, and workload.[21] Resident physicians in primary care specialties comprise a substantial fraction
of the healthcare workforce and regularly utilize the CPOE. Throughout training, resident
physicians are subject to different work environments and variable workload intensities
across the various subspecialties pertaining to a specific residency program. These
circumstances are unique to resident physicians, as compared with established medical
staff working under less variable conditions or who may have subspecialized functions
within an institution. In this study, we explore salience rates among internal medicine
resident physicians at two integrated healthcare institutions.
Objective
Our objective is to retrospectively evaluate the association between residence year
and drug-alert prescriber response. We hypothesize that residents further along in
the program have lower salience rates, possibly reflecting long-term desensitization.
Methods
Institutions
The University of Pennsylvania Medical Center (UPMC) Pinnacle is a healthcare system
which employs over 2,900 physicians across seven acute care hospitals and 160 ambulatory
clinic sites. In 2014, UPMC Pinnacle adopted the Epic (Epic Systems Corporation, Verona,
Wisconsin, United States) EHR system along with CPOE, and has achieved Healthcare
Information Management and Systems Society (HIMSS, Chicago, Illinois, United States)
Electronic Medical Record Adoption Model Stage-6 certification. The UPMC Pinnacle
uses the drug information vendor First Data Bank (Hearst Health Network, New York,
New York, United States) to provide drug information for CDS.
The MetroHealth System (MHS) is an academic healthcare system employing over 550 physicians,
and more than 350 physicians in training across 21 health centers, 4 emergency departments,
and 13 schools. In 1999, the MHS adopted the Epic (Epic Systems Corporation) EHR system
along with CPOE. MHS has achieved Healthcare Information Management and Systems Society
(HIMSS) Electronic Medical Record Adoption Model Stage-7 certification. MHS uses the
drug information vendor MediSpan (Wolters Kluwer Clinical Drug Information, Hudson,
Ohio, United States) to provide drug information for CDS.
Both institutions offer dedicated classrooms for in-person and computer-based EHR
training at the start of a residency program. Training lasts approximately 2 weeks,
following which user proficiency is tested. Ongoing refresher training sessions are
mandatory on a regular basis with computer-based EHR sessions. Additionally, resident
physicians are involved in advisory committees that discuss CDS tools. Both institutions
employ Chief Medical Information Officers who are supported by pharmacy informatics
staff for constant monitoring and evaluation of drug alerts.
Residency Programs
Internal Medicine Residency Programs consist of 3 years of training, postgraduate
year 1 (PGY-1) to postgraduate year 3 (PGY-3). The training program at UPMC Pinnacle
consists of 20 PGY-1 (interns), 14 PGY-2, and 13 PGY-3 residents. The MHS consists
of 34 PGY-1, 23 PGY-2, and 22 PGY-3 residents ([Table 1]).
Table 1
Drug alert and salience rates across resident training years at the two institutions
|
Both institutions
|
|
Year
|
Residents
|
Alerts (3 months)
|
Changes/removals
|
Salience (%)
|
Alerts/resident/month
|
|
PGY-1
|
54
|
22,290
|
2,589
|
11.6
|
138
|
|
PGY-2
|
37
|
19,530
|
2,043
|
10.5
|
176
|
|
PGY-3
|
35
|
10,804
|
965
|
8.9
|
102
|
|
Total
|
126
|
52,624
|
5,597
|
10.6
|
158
|
|
p < 0.05 for decrease in salience among the three years of residency
|
|
UPMC Pinnacle
|
|
PGY-1
|
20
|
5,558
|
1,053
|
18.9
|
93
|
|
PGY-2
|
14
|
6,177
|
954
|
15.4
|
147
|
|
PGY-3
|
13
|
3,839
|
400
|
10.4
|
98
|
|
Total
|
47
|
15,574
|
2,407
|
15.4
|
110
|
|
p < 0.05 for decrease in salience among the 3 years of residency
|
|
MHS
|
|
PGY-1
|
34
|
16,732
|
1,536
|
9.2
|
164
|
|
PGY-2
|
23
|
13,353
|
1,089
|
8.15
|
193
|
|
PGY-3
|
22
|
6,965
|
565
|
8.11
|
105
|
|
Total
|
79
|
37,050
|
3,190
|
8.6
|
156.3
|
|
p < 0.05 for decrease in salience from PGY-1 to PGY2 or PGY-3. p >0.05 for difference in salience from PGY-2 to PGY-3
|
Abbreviations: MHS, the MetroHealth System; PGY, postgraduate year; UPMC, the University
of Pennsylvania Medical Center.
Drug Alerts and Responses Elicited
Among the 13 drug-alert categories, this study looked at the most common types of
drug alerts, corresponding to the following categories: duplicate medications, drug
interactions and compatibility issues, allergies, and misadministrations in terms
of dosage and frequency. Resident physicians had three different options to respond
to each drug alert ([Table 2]): (1) adjusting medication settings, such as dose or interval of administration;
(2) cancelling the order, or (3) overriding the drug alert and continue placing the
order with original settings.
Table 2
Categories of responses to drug alerts
|
Categories of drug-alert response
|
Salience type
|
|
Adjusting the medication setting
|
Salience type—adjusting the medication setting including dose or interval administration
|
|
Cancel the order
|
Salience type—allows residents to essentially restart by cancelling the original order
|
|
Overriding the drug alert
|
Nonsalience response, choosing to ignore the clinical decision support recommendation
|
Study Design
Investigators at each academic healthcare institution performed a retrospective cross-sectional
review of drug alerts encountered by resident physicians from their corresponding
Internal Medicine Residency Programs. The EHR database (Epic Systems) was queried
to collect drug-alert information corresponding to Internal Medicine residents at
the UPMC Pinnacle and MHS, from December 2018 through February 2019. Multiple variables
were collected and classified, including number of drug orders placed per year of
residency training, number of alerts triggered, and response elicited.
Analysis
The primary endpoint analyzed was salience rate, defined as percentage of drug alerts
that elicited either removal or change of order parameters (i.e., change of dose,
frequency, or timing) as opposed to overriding the drug alerts. Comparisons were made
across residency training levels and between organizations for the number of alerts
as per residency training year and number of alerts generated as per resident per
month. Salience rates for alerts were compared across each residency program and between
organizations.
Statistical analyses were performed utilizing SPSS software (version 1.0.0.1327; IBM)
to calculate salience and override rates as per year of residency, in addition to
correlations and associations and descriptive statistics for percentages, frequencies,
and rates. Chi-square test was conducted to look for differences in the percentage
of removed/changed orders among the three levels of PGY. To compare the means of number
of removed/changed orders, one-way analysis of variance (ANOVA) was used to detect
differences among the levels of PGY. To determine which groups were different from
one another, Bonferroni's method was utilized.
Institutional Review Board (IRB) was consulted as per guidelines at each institution,
an institutional review board (IRB) protocol approval was obtained as required by
each institution. Program director approval was obtained from each residency program.
Results
Drug-Alert Exposure
A total of 126 residents were exposed to 52,624 alerts over a 3-month period. At UPMC
Pinnacle, 15,574 alerts were generated corresponding to 47 residents (average of 331
alerts per resident during the 3-month study period), and at MHS 37,050 alerts were
generated corresponding to 79 residents (average of 469 alerts per resident during
the 3-month study period). The difference in mean values of number of alerts per resident
between the two institutions was significant (p < 0.01).
Salience Rates
Overall, 54 PGY-1 residents were exposed to 22,290 alerts, eliciting 2,589 changes
or removals, corresponding to a salience of 11.6%; 37 PGY-2 residents were exposed
to 19,530 alerts, eliciting 2,043 changes or removals, corresponding to a salience
rate of 10.46%; 35 PGY-3 residents were exposed to 10,804 alerts, eliciting 965 changes
or removals, corresponding to a salience of 8.9%. The mean number of drug alerts seen
per resident per month were 138, 176, and 102, for PGY-1, PGY-2, and PGY-3, respectively
([Table 1]).
The University of Pennsylvania Medical Center Pinnacle
Twenty PGY-1 residents were exposed to 5,558 alerts, eliciting 1,053 changes or removals,
corresponding to a salience rate of 19%. Fourteen PGY-2 residents were exposed to
6,177 alerts, eliciting 954 changes or removals, corresponding to a salience rate
of 15%. Thirteen PGY-3 residents were exposed to 3,839 alerts, eliciting 400 changes
or removals, corresponding to a salience rate of 10%. The mean number of alerts seen
per resident per month was 93, 147, and 98, for PGY-1, PGY-2, and PGY-3, respectively
([Table 1]).
The MetroHealth System
Thirty-four PGY-1 residents were exposed to 16,732 alerts, eliciting 1,536 changes
or removals, corresponding to a salience rate of 9.2%. Twenty-three PGY-2 residents
were exposed to 13,353 alerts, eliciting 1,089 changes or removals, corresponding
to a salience rate of 8.2%. Twenty-two PGY-3 residents were exposed to a total of
6,965 alerts, eliciting 565 changes or removals, corresponding to a salience rate
of 8.1%. The average number of alerts per resident per month was 164, 193, and 105
for PGY-1, PGY-2, and PGY-3, respectively. Over a 3-month period, the average number
of alerts seen per resident at MHS was 468 which was 41% higher than the average of
331 seen at UPMC Pinnacle ([Table 1]).
At UPMC Pinnacle, saliency rates were significantly different among PGY-1, PGY-2,
and PGY-3 (p < 0.001). At MHS, saliency rates were significantly higher for PGY-1 residents (p < 0.05), as compared with PGY-2 and PGY-3, but not significantly different when comparing
PGY-2 and PGY-3 (p > 0.05). In the overall sample, including both institutions, salience rate was significantly
different among all three groups (p < 0.001).
Discussion
We conducted a cross-institutional retrospective study where we observed and compared
drug-alert rates and salience between PGY1, PGY-2, and PGY-3 at two different institutions.
We demonstrated a significant variation in response rates between different residency
training years. The number of triggered drug alerts was significantly fewer among
PGY-3 compared with PGY-1 and PGY-2. This trend was insignificant when comparing PGY-2
and PGY-3 at MHS. Overall, PGY-3 residents were exposed to 45% fewer alerts than PGY-2
and 26% fewer than PGY-1.
A higher number of alerts-per-resident was seen in PGY-2 as compared with PGY-1 and
PGY-3. This phenomenon may reflect the increased clinical responsibilities of PGY-2
during the first 6 months of the academic year (July–December) where PGY-1 learnt
to utilize the EHR and CPOE and consequently placed fewer orders. An additional consideration
is that PGY-2 may have heavier clinical workloads, typically evaluating more patients
and placing more orders as compared with junior residents.
PGY-1 were more likely to remove or change orders than senior residents based on drug-alert
exposure. This tendency of decreasing salience from PGY-1 to PGY-3 despite a variable
number of alerts-per-resident in each year, may be multifactorial. While the trend
may suggest long-term desensitization, as PGY-2 and PGY-3 were more likely to override
a drug alert, it may also reflect an increased experience in PGY-2 and PGY-3 utilizing
the CPOE: PGY-2 and PGY-3 may better identify inappropriate drug alerts, or drug alerts
shown to these groups may be more likely inappropriate than those triggered by orders
from less knowledgeable residents (PGY-1), and could be therefore more likely overridden.
It should also be noted that the tendency of decreasing saliency from PGY-2 to PGY-3
was minimal in MHS as compared with UPMC Pinnacle.
Comparing results between the two institutions, residents at MHS showed a 41% higher
number of alerts-per-resident which was associated with a 42% decrease in overall
salience rate. This finding may support two hypotheses. First, there may be a degree
of drug-alert fatigue due to repetitive or overall higher numbers of alerts at MHS
which was evident when comparing the two institutions, and second, resident physicians
may be subject to long-term desensitization while progressing through residency. Due
to substantial differences in both institutions, however, including vendor, culture,
and training, among others, a direct comparison of salience rates can be misleading.
By the final years of residency, trainees of primary care specialties have likely
been exposed to a large number of medication safety alerts, which raises a concern
for possible drug-alert desensitization which has been previously suggested as a cause
of decreased salience in medical providers.[21]
[22]
[23]
[24]
[25]
[26]
Some authors who have included resident physicians while investigating salience have
found variable response rates. Knight et al, for example, found that alerts were more
likely to be overridden when encountered by residents,[27] which seems inconsistent with findings reported by Weingart et al[24] who reported that residents were less likely to override medication alerts, and
Long et al suggested that physicians with longer years in practice were more resistant
to innovation and less likely to accept CDS reminders.[25] However, most studies analyzing salience rates during residency do not account for
many variables that may affect drug-alert experience from a residency standpoint,
including year of training, specialty, and workload. These variables should be considered
when comparing salience not only between residents, but between different providers.
As previously shown, the UPMC Pinnacle demonstrated an inverse relationship between
salience and year of training. At MHS, this trend was demonstrated only when comparing
PGY-1 to PGY-2 or PGY-3, but not significantly different between PGY-2 and PGY-3.
PGY-3 at MHS was exposed to a 54% lower number of alerts per resident per month (105)
than PGY-2 (193) ([Table 1]). Despite this difference, however, salience was similar. Based on a premise of
drug-alert fatigue, where a higher drug-alert burden leads to lower salience rates,
PGY-3 at MHS would have been expected to have a higher salience rate than PGY-2, the
fact that this number did not change may suggest long-term desensitization.
Also noted was an increased number of alerts per resident in PGY-2 at both institutions.
We believe that this is secondary to two main factors. First, PGY-1 residents place
relatively fewer medical orders in their first months of residency, while PGY-2 and
PGY-3 have relatively more elective rotations where the number of orders placed is
generally lower than in core rotations (i.e., medical wards and intensive care unit).
Residents at MHS were exposed to an average to five alerts per day, while those at
UPMC Pinnacle saw approximately four alerts per day. Dexheimer et al analyzed results
from 4,575 providers over a 24-month period showing that providers exposed to 49 alerts/day
showed maximum saliency. However, this sample included various types of providers
including attending physicians and fellows of different specialties. When analyzing
only residents, however, Dexheimer et al found a relatively constant salience trend
overtime since the beginning of residency which contrasts results from our study.[4] Additionally, Dexheimer et al found a higher alert/day rate which could be related
multiple variables including patient background (e.g., pediatric- versus nonpediatric-based
healthcare institution) and out-of-box vendor-alert customization differences.
Other authors have found a decreasing saliency with increasing numbers of alerts per
provider per day, suggesting drug-alert fatigue.[4]
[7]
[25] While this trend may be secondary to cumulative exposure to drug alerts, another
possibility may be that senior residents were more experienced utilizing the CPOE,
thus the smaller number of alerts they triggered were most likely inappropriate and,
therefore most commonly overridden. Steele et al compared different types of drug-laboratory
interactions in the outpatient setting, with the majority of orders being placed by
faculty members, rather than residents, showing that providers would generally continue
ordering a medication despite drug alerts.[8]
In a retrospective study, Zenziper Straichman et al found a drug-alert acceptance
rate (salience) of 5.3% (p < 0.001) from providers at an internal medicine department. While their sample included
faculty members, the majority was composed of resident physicians (64 out of 92).
In their prospective study that included two internal medicine departments, Zenziper
Straichman et al reported a drug-alert acceptance rate of 4.2%, of which 89.3% corresponded
to orders placed by residents and 10.7% to faculty members.[26] Similar to Zenziper Straichman et al, Knight et al found a salience of 4%, less
than half of what was found in our study (11%). Knight et al also found that alerts
were more likely to be overridden when encountered by residents or providers younger
than 40 year of age[27]; however, drug-alert burden was not taken into consideration which may partly contribute
to lower salience rates among resident physicians. Unlike Knight et al, Weingart et
al found that residents were less likely to override medication alerts,[24] and Long et al showed that physicians with longer years in practice were less likely
to accept reminders and suggested that older providers may be more resistant to innovation.[25] When comparing salience rates among different providers, multiple variables should
be taken in consideration including number of alerts per provider per unit of time,
work hours, overall workload, and other factors which may otherwise confound direct
comparisons.
Systematic approaches for optimizing drug alerts have been suggested by other authors
to improve salience rates. Saiyed et al highlighted the importance of reviewing reported
data and receiving feedback from end users who can offer suggestions for improving
the utility of drug alerts.[15] Stutman et al proposed an iterative approach to modify drug-alert display options
by tracking their frequency and elicited responses in providers.[28]
Nevertheless, inappropriate drug alerts overriding should also be considered, which
may occur due to a variety of reasons such as cognitive overload, inadequate understanding
of drug alerts, or desensitization,[4] all of which are possible during residency training.
The overall number of drug alerts-per-resident in the 3-month period was 42% higher
in MHS, as compared with UPMC Pinnacle, which was associated with a 41% decrease in
overall salience rates, suggesting that there could be some degree of drug-alert fatigue
due to excessive or repetitive drug alerts; a phenomenon that has been suggested in
prior studies,[4]
[7]
[20]
[21]
[26] may be associated with cognitive overload.[21] Ancker et al conducted a retrospective cohort including 112 ambulatory primary care
providers and suggested that alert fatigue can be associated with cognitive overload;
however, they found no associations with workload or evidence of time-sensitive desensitization.[21] Our study was limited because we did not measure time. Nonadherence to drug alerts,
manifested as high override rates, has been reported in 49 to 96% of alerts in other
samples,[15] and may be secondary to multiple causes including poor implementation or acceptance
of CDS tools, drug-alert fatigue, and possibly long-term desensitization. Continued
optimization of drug alerts is needed to decrease the number of inappropriate alerts.
Communication with medical providers, including resident physicians, is necessary
to recognize obstacles that may lead to nonadherence to CDS tools.
Limitations
This study had some limitations. It was restricted to internal medicine residency
programs. Other specialties have different levels of clinical workload and patient
demographics which may impact drug-alert response rates. The study compared two different
residency programs with multiple variables that were not considered including differences
in culture, subspecialty rotations, and drug information vendors, among other factors.
The study only included data from 3 months, a 3-year prospective study would allow
exploring salience trends, and responses to new CDS implementations throughout residency
training. Finally, the study lacked assessment of alert appropriateness, such as recurrent
inappropriate duplicate alerts to single users despite prior reasonable overrides.
Conclusion
This is one of the first cross-institutional studies examining drug-alert rates and
comparing salience rates between years of medical training and medical experience
between two academic institutions utilizing the same EHR. We demonstrate a drug-alert
salience of 11% in 126 trainees from two large academic-based residency programs,
with significantly different response rates between different years of training. Override
rates increased progressively from junior to senior years of residency training. This
trend, along with the lack of a significant difference in salience between PGY-2 and
PGY-3 at MHS despite nearly half the rate of drug alerts-per-resident, favors long-term
desensitization. However, other factors should also be considered.
Resident physicians may be at risk for long-term desensitization to drug alerts from
cumulative exposure and drug alert fatigue from repetitive exposure which may lead
to inappropriate overrides and medical events. Continued optimization of CDS tools
and appropriate training on CPOE systems, starting from medical school and continuing
through residency, could hypothetically decrease the risk of inappropriate overrides
and, consequently, serious medical events.
Clinical Relevance Statement
Clinical Relevance Statement
Drug alerts and clinical decision support tools have demonstrated decreased risk of
medical events including medication misadministration. High override rates in resident
physicians may be secondary to various factors, including adequacy of drug alerts,
nonadherence, drug-alert fatigue, and chronic desensitization. Identifying these obstacles
will facilitate strategies for drug-alert optimization.
Multiple Choice Questions
Multiple Choice Questions
-
Which of the following has/have been proposed as a cause for increasing override rates?
Correct Answer: The correct answer is option e.
-
Improvement of saliency rates in physician response to medication alerts, may lead
to
Correct Answer: The correct answer is option e.
