Keywords
adverse drug event - medication errors - patient safety - electronic health records
- error management and prevention
Background and Significance
Background and Significance
Electronic health record (EHR) systems can be equipped with features to prevent prescribing
errors. Forcing functions like order sets and sentences are designed to prevent prescribing
errors at the point of initiating medication orders, and active interruptive clinical
decision support (CDS) can prevent errors from being submitted or from reaching the
patient if interruptive alerts occur in verification processes.[1] Despite these features and the rapid growth of EHR systems, prescribing errors are
still prevalent and have the potential to result in patient harm.[2]
[3] However, prescribing errors can be difficult to detect, as they are often not voluntarily
reported unless they result in harm. The use of trigger tools, which are defined as
“a data element present within the health record which may represent the presence
of an adverse drug event (ADE) or error which has occurred,[4]” is considered one of the most effective methods of adverse event detection,[4]
[5]
[6]
[7] identifying events not detected through other techniques. Using an EHR system improves
the efficiency of the trigger tool technique by enabling the selective query of data
elements that constitute a trigger (e.g., selected medications and laboratory values).
While triggers are traditionally designed to detect patient harm, the information
gathered from the investigations of near-miss safety events can also be significant
in improving patient safety.[8]
[9]
[10]
[11] Koppel et al revised an existing trigger, abrupt medication stop, to detect no harm,
near-miss prescribing errors from rapidly discontinued medication orders.[12] In their study, Koppel et al provided substantial evidence to support their hypothesis
that medication orders stopped within 120 minutes of original submission are often
prescribing errors. However, their approach of having an individual dedicated to monitoring
computerized prescription order entry (CPOE) systems in real time for rapidly discontinued
medication orders and following up by interviewing prescribers to determine if an
error occurred, has practical challenges for use as a detection method in routine
practice.
Objectives
This study complements the research of Koppel et al by assessing retrospective techniques
for the detection of prescribing errors in rapidly discontinued medication orders,
which would allow for data to be reviewed regularly (e.g., weekly or monthly) without
interrupting patient care practices. The aims of this study were (1) to develop methods
for retrieving rapidly discontinued medication orders from pediatric EHR data, (2)
to assess the ability to retrospectively detect prescribing errors from the retrieved
EHR (similar to the application of using trigger tools to detect ADEs), and (3) to
characterize the identified errors and identify practical applications of the approach
for detecting prescribing errors.
Methods
Study Design
The study was reviewed and approved by the hospital's institutional review board.
A random number generator was used to identify 28 days during 2012 in which medication
orders from the EHR system would be queried. To address the potential of the “July
effect” on errors, the random number generator was repeated until only 2 days from
July were selected.[13] The first developed query identified all orders within the 28-day span, and a subsequent
one extracted orders that were altered within 120 minutes of initial entry. All inpatient
and outpatient medication orders within this timeframe on these days were included
in the study. Inpatient orders were defined as those to be administered in the hospital,
incorporating admitted inpatients and those visiting the hospital for outpatient clinical
services. Outpatient orders were defined as prescription medications to be administered
outside the hospital. Fields collected in the data query were medical record number,
patient medical service, primary diagnosis, unique order identification number, date/time
of original order entry, name and position of prescriber, medication, therapeutic
class, date/time of order discontinuation, and name and position of the person who
discontinued the original order.
Prescribing error was defined as “a failure in the prescription writing process that
results in a wrong instruction about one or more of the normal features of a prescription”
where “normal features” of a prescription would include, but not be limited to, the
identity of the recipient; the identity of the drug; the formulation and dose; and
the route, timing, frequency, and duration of administration.[14] The positive predictive value (PPV) of the detection method was calculated as the
fraction of rapidly discontinued orders that were determined to likely be errors.
This method of calculating PPV was used previously.[12]
Similar to Koppel et al, we excluded orders discontinued within 1 minute of the initial
submission to avoid immediately self-corrected mistakes, typographic, or selection/drop-down
menu picking errors.[12] Because medication orders can be stopped by using various CPOE functions, the following
actions were included in the query: cancel, complete, delete, discontinue, and modify. Initial review indicated that data captured by the order action modify largely represented routine pharmacy tasks that do not represent prescribing errors,
such as dose rounding per hospital policy or adding the appropriate diluent. Therefore,
modify order actions were removed from subsequent analyses. Also, the query returned no
orders attributed to the order action delete. Complete is an action often used by mid-level providers and pharmacists, it essentially has
the same function as discontinue yet does not require a physician cosignature. Therefore, the final data set contained
only medication orders that were canceled, completed, or discontinued within 1 to
120 minutes of original entry.
Setting
The study hospital is a tertiary referral center for children with pediatric malignancies.
It does not offer emergency services or have a neonatal intensive care unit. The hospital
provides inpatient and outpatient medications for patients, and all prescriptions
are entered through CPOE. For inpatient and outpatient services, the hospital has
clinical pharmacists who are responsible for optimizing all drug therapies, generating
all intravenous nutrition orders, and providing pharmacokinetic consultation services.
The study hospital has fully implemented an EHR system with CPOE (Millennium system,
Cerner Corporation, Kansas City, Missouri, United States) for all aspects of inpatient
and outpatient care, including orders, documentation, laboratory, and pharmacy.[15] During the study period, the hospital had active and passive CDS systems. Interruptive
CDS alerts for drug–drug interactions, drug–allergy, and dose range checking were
active for the study period. For passive CDS, roughly 3,000 order sets were active
and more than 30,000 order sentences were available to guide prescribing. To manage
alert fatigue caused by interruptive CDS alerts, a team made proactive decisions in
2005, such as limiting drug interaction alerts to the highest level (based on the
Multum content database) and disabling duplicate therapy (i.e., therapeutic class
duplication) alerts.
Data Analysis
Two pharmacists independently reviewed the data by examining the queried data files
and also seeking additional details from EHR patient chart records to determine whether
the discontinued orders were due to a prescribing error. The orders were categorized
as most likely a prescribing error, most likely not a prescribing error, or not enough information to determine whether an error occurred. Discovered errors were further characterized by clinical significance as being either
potentially clinically significant or not significant through independent review by two study authors. Clinical significance was defined
as likely to have caused harm if it reached the patient. Percentage agreement and
Cohen's kappa were used to evaluate interrater reliability among the pharmacists reviewing
the data. For the orders that were identified as likely errors, the interruptive CDS
alert log files were reviewed to determine whether an alert designed to prevent the
prescribing error occurred during the ordering process.
To assess the performance of the error detection method throughout the 2-hour timeframe,
the data were divided into 15-minute intervals, and percentage of errors with confidence
intervals was calculated for each segment. Errors discovered during the analysis were
cross-referenced with records from the hospital's voluntary electronic event reporting
system.
To identify potential system improvements, the data were analyzed by prescriber-type
and established error categories.[16] To compare the prevalence of errors across prescriber types, error proportions were
calculated by dividing the number of observed errors by the total number of orders
entered during the study period (i.e., not limited to those discontinued within 120
minutes) and expressed as an error rate per 1,000 orders. Pearson's chi - square was
used to compare rates of errors across prescriber types.
Results
A total of 305 medication orders (corresponding to 176 patients) that were canceled,
completed, or discontinued within 120 minutes of being submitted were reviewed as
potential prescribing errors. The most common primary diagnosis of the patients was
acute lymphocytic leukemia (n = 73, 23.9%). Analgesics were the most common class of medication associated with
rapid discontinuations (n = 46, 15.1%). The average time needed to review each order was 2 minutes. Interrater
agreement across the three determination options (e.g., most likely a prescribing error, most likely not a prescribing error, and not enough information to determine whether an error occurred) was initially 65% (κ = 0.46), which corresponds to a moderate level of agreement.[17] However, the reviewers reached 100% consensus after meeting to discuss initial assessments.
Disagreement among reviewers was most often because one reviewer did not initially
notice a key piece of information in the health record. Once this was brought to the
other reviewer's attention, consensus was reached for all orders.
Roughly half (n = 147, 48%) of the canceled, completed, or discontinued orders were determined to
be most likely a prescribing error ([Table 1]). One hundred forty-three of the orders were for inpatient contexts. Most of the
prescribing errors were self-corrected (121, 82.3%), pharmacists corrected 7 (4.7%)
errors during their review processes, and the remaining 19 errors were corrected by
clinicians other than the original prescriber (e.g., an attending physician correct
an error made by a physician in training). Analysis of the rate of errors at 15-minute
time points indicated that the trigger was most predictive for orders stopped within
the first 90 minutes (PPV = 0.54 versus 0.21 for 91–120 minutes). Most of the identified
prescribing errors occurred within 15 minutes of being submitted (n = 88, 59.9%), and more than three-fourths occurred within the first 45 minutes (n = 121, 77.6%). None of the detected errors had been reported to the hospital's voluntary
electronic event reporting system.
Table 1
Reasons for discontinuation of medication orders within 120 minutes of original submission
|
Reason for discontinuation
|
0–15 min (n)
|
16–30 min (n)
|
31–45 min (n)
|
46–60 min (n)
|
61–75 min (n)
|
76–90 min (n)
|
91–105 min (n)
|
106–120 min (n)
|
Total (N)
|
|
Error
|
88
|
16
|
10
|
17
|
6
|
6
|
3
|
1
|
147
|
|
Nonerror
|
9
|
4
|
7
|
8
|
2
|
2
|
5
|
14
|
51
|
|
Undetermined
|
39
|
26
|
9
|
5
|
7
|
5
|
7
|
9
|
107
|
|
Error % (95% CI)
|
65 (55–71)
|
34 (22–50)
|
38 (21–59)
|
56 (38–74)
|
40 (17–67)
|
46 (20–74)
|
20 (5–49)
|
4 (0–23)
|
48 (43–54)
|
Abbreviation: CI, confidence interval.
Note: Error corresponds to cases determined to be most likely a prescribing error; Nonerror to most likely not a prescribing error; and Undetermined to not enough information to determine whether an error occurred.
Independent review of the errors to categorize them as being either potentially clinically significant or not significant yielded an initial 82.3% (κ = 0.42), agreement between the two reviewers. After meeting
to discuss individual assessments, 100% consensus was achieved. In all, 15.6% (n = 24) of the errors were determined to be potentially clinically significant had they reached the patient. Included in these were significant opioid analgesic
overdoses (e.g., 2.3 times hydromorphone overdose) and two underdoses (e.g., changing
a pain crisis patient's medication from morphine to hydromorphone). Among the potentially
clinically significant errors, the lag times between initial order submission and
discontinuation were not normally distributed (Kolmogorov–Smirnov = 0.24, p < 0.001), with a median time of 10 minutes (range = 1–107). Of the potentially significant
errors, 54% (n = 13) of the orders were discontinued within 15 minutes.
Of the orders that were identified as likely errors, three relevant interruptive CDS
alerts occurred on three separate orders during the prescribing process, but these
alerts did not prevent the error. Two of the alerts were for previously documented
patient drug allergy histories (i.e., a penicillin allergy alert was documented in
the chart, yet the medication was still ordered), and one alert occurred because the
medication dose was out of the dosing range.
The number of errors detected in the study was compared with the distribution of overall
prescriptions written by practitioner type at our hospital ([Table 2]). A total of 16,687 medication orders were submitted during the study period, which
corresponded to error rates ranging from 2.6 to 15.9 errors per 1,000 orders ([Table 2]). Nurse practitioners and physician assistants made the most prescribing errors
and submitted the most orders ([Table 2]). However, physicians in training had a higher error rate than did all other types
of prescribers (p < 0.001). Most of the 147 identified prescribing errors were categorized as duplicate
orders (n = 45, 30.6%) ([Table 3]).
Table 2
Prescribing errors by prescriber type (N = 147)
|
Prescriber type
|
% of errors
|
% of overall prescribing[a]
|
Error rate/1,000 orders (95% CI)
|
|
Nurse practitioner/Physician assistant
|
43
|
46
|
9.2 (7.1–11.6)
|
|
Physician in training
|
37
|
20
|
15.9 (11.9–20.7)[b]
|
|
Physician
|
16
|
27
|
5.8 (3.7–8.5)
|
|
Other (e.g., Pharmacist)
|
4
|
7
|
2.6 (0.5–7.5)
|
Abbreviation: CI, confidence interval.
a % of institution's overall medication orders written by staff during the study period.
b Larger error rate compared with that of other prescriber types (p < 0.001).
Table 3
Frequency of different types of prescribing errors (N = 147)
|
Error type
|
n
|
%
|
|
Duplicate order
|
45
|
31
|
|
Wrong route
|
22
|
15
|
|
Wrong drug
|
21
|
14
|
|
Ordered incorrectly
|
14
|
10
|
|
Wrong dosage form
|
9
|
6
|
|
Underdose
|
8
|
5
|
|
Overdose
|
5
|
3
|
|
Wrong frequency
|
4
|
3
|
|
Wrong patient
|
1
|
1
|
|
Other
|
18
|
12
|
Discussion
Similar to previous research, our study revealed that many rapidly discontinued medication
orders are likely prescribing errors.[12] Roughly half of the reviewed orders were determined to be errors (48%) and error
rates were as high as 1.6% for certain prescriber types.
The work by Koppel et al was important foundational research that supported the hypothesis
that medication orders discontinued within 2 hours of original entry are often errors.
However, the prospective methods of their study would require someone to regularly
interrupt prescribers to discuss medication orders and to determine whether the discontinued
order was due to an error, which could have a variety of detrimental effects.[18]
[19]
[20] Their method also requires someone to monitor the EHR system in near real time to
determine whether an order is discontinued within 120 minutes. Therefore, the challenges
of sustaining the prospective approach and potential patient safety risks of routinely
interrupting prescribers of the may hinder the method's practicality for regular use.[18]
[19]
[20] Our research has demonstrated that similar results can be achieved by using automated,
trigger tool–type, retrospective methods, which are potentially more efficient and
pose less risks. Our results also cross-validate the method for a pediatric patient
population, as the initial research was conducted in an adult setting.
Given that none of the detected errors were reported to our hospital's voluntary reporting
system, our method provided information for discovering improvement opportunities
that would have likely been missed otherwise. This finding compares similarly with
other research that has highlighted the low reporting rates for prescribing errors.[21]
[22] It also empirically reinforces previously expressed sentiments that multiple detection
methods are needed to fully understand the extent of errors and adverse events in
a health care setting.[4]
[23] Further, only 7 of 147 (5%) of the orders were discontinued and corrected by pharmacists
during the standard pharmacy order verification process. Therefore, this method appears
to be finding prescribing errors that are not simply being recorded by other means
such as intervention databases often maintained by pharmacists.
Examining the trend of errors over the time from original order submission revealed
a drop-off in the rate of errors after 90 minutes. Approximately 55% of the prescribing
orders before 90 minutes were errors versus 21% in the 91 to 120 minute segment. Also,
47% of the order discontinuations evaluated in the study occurred within 15 minutes
of initially being submitted. The orders within this time period also composed most
of the errors detected in the study. Furthermore, the rate of errors within the first
15 minutes was 15% higher than the rate overall, and most of the potentially significant
errors also occurred within this timeframe. On the basis of our data, particular attention
should be focused on medication orders canceled, completed, or discontinued within
15 minutes, yet extending this timeframe to 90 minutes would still yield reasonable
rates of identified errors.
In contrast to findings in previous research that has used “abrupt medication stop”
as a trigger to detect patient harm,[4]
[5]
[24]
[25]
[26] the errors discovered in our study are “near-miss” patient safety events.[27] That is, these prescribing errors did not reach patients and are distinct from ADEs
that result in harm, which are the primary focus of trigger tools. Although these
errors were caught and corrected before reaching the patients and only 15% of them
were potentially serious, there is value in understanding these events to inform process
improvement in CPOE and medication use more broadly. Moreover, near-miss safety events
have been shown to occur up to 100 times more than adverse events.[21]
Other contextual details of the detected errors should reveal additional improvement
opportunities. For example, potentially clinically significant errors were identified
in our study, including dosing errors for opioid analgesics and chemotherapy, which
are high-alert medications. This information could prompt a review of prescribing
practices and policies pertaining to high-alert medications. Differences in the number
of prescribing errors by prescriber type can highlight systematic performance gaps,
prompting prescribing training and other interventions. Improvement efforts can also
be directed by examining the most frequent types of errors, which were duplicate therapy
orders in our study. An example of a direct practical application of this particular
result is that it has prompted our hospital to reexamine its use of interruptive duplicate
therapy CDS alerts. Because we found that three interruptive CDS alerts did not prevent
prescribing errors, this event detection approach may also help better understand
the effectiveness of interruptive alerts.
Preventing patient harm is the ultimate goal of eliminating prescribing errors, but
such errors also place an unnecessary demand on resources. The moment a medication
order is submitted, structured work cycles begin. Pharmacists begin checking orders
for correctness, pharmacy technicians prepare medications, and inpatient orders are
delivered and administered. Errors in prescribing that are caught and changed can
result in double work and the loss of often costly medications. Thus, detecting prescribing
errors can provide feedback on efficiency. Inefficiency from prescribing errors becomes
particularly significant considering the potential number of errors over an entire
year versus our 28-day study period. To emphasize this point, we queried all of the
medication orders from our study year (2012) that would qualify as rapidly discontinued,
completed, or canceled within 120 minutes. The queries returned 5,837 orders, which
equates to 2,685 prescribing errors given our current result of a 48% percentage of
errors. Extrapolating this rate from an initial validation data set to a larger one
has been done in a similar context and used to evaluate the performance of interventions
to prevent wrong patient orders.[27]
[28]
Limitations and Future Directions
The initial interrater agreement for determining if the reviewed medication orders
were errors was relatively low (i.e., agreement was 65%, Cohen's kappa, 0.46). Despite
both reviewers researching full consensus upon discussing their determinations, this
initial rate of disagreement threatens the reliability of the measure. One explanation
for this disagreement is that the reviewers were not the initial prescribers and that
the design of the study was retrospective. Prospective designs for detecting medication
errors have advantages over retrospective ones in their ability to be certain if an
order was actually a mistake and to clarify context in which an error occurred. For
example, one recently published paper utilized a prospective design to study reasons
prescribers discontinued medication orders. While this study yielded illuminating
results, the retrospective design described on our study has the potential to be used
more regularly for continuous data collection to inform improvement.[29] Our study should be repeated, with a keen interest on increasing the reliability
of the measure, possibly through additional methods to selectively target critical
EHR data needed to determine if an order's discontinuation was due to error. Also,
given the unique focus of the study hospital, the results of our study may not be
generalizable to other settings, and further research should validate this error detection
method in different clinical settings.
The feasibility of reproducing this method in health care settings that do not use
the Cerner EHR could also be explored. Although the average time to determine whether
a rapidly discontinued medication order was an error was relatively low at roughly
2 minutes, future research could investigate ways to fully automate prescriber error
identification. Additionally, we excluded the review of modify order actions because many of these were clinically appropriate actions taken by
pharmacists, and it may be possible to more selectively retrieve modified orders so
they could be assessed as potential prescribing errors. Although our queries did not
return any orders with the order action delete, other settings may use this action more widely, which may represent prescribing
errors. Future research could also use this method to test the effectiveness of interventions
to reduce errors.
Conclusion
Our research demonstrated that it is possible to identify prescribing errors retrospectively
from electronically submitted medication orders that were rapidly discontinued in
a pediatric patient population. Given that this measure was originally developed using
prospective methods that require interrupting prescribers to discuss order details,
validating it using retrospective methods enables it to be used more regularly. Extending
the practical utility of this error detection measure allows for gathering more critical
information that can assist in determining the root causes of prescribing errors,
tracking performance, and identifying and evaluating interventions to improve prescribing
systems and processes.
Clinical Relevance Statement
Clinical Relevance Statement
This study demonstrated that prescribing errors can be systematically identified in
electronic health record data by targeting medication orders that were rapidly discontinued.
Previous research has suggested this is possible through prospective designs, but
this study extended the use of this technique to retrospective methods. The identification
of errors is critical to improving health care systems, and the methods described
in this article may enable administrators and researchers identify errors that were
previously unknown.
Multiple Choice Question
-
Which method of detection is most likely to identify harmful patient safety events?
Correct Answer: The correct answer is d. While each method of detection may identify harmful patient
safety events, trigger tool's effectiveness is largely predicated on the occurrence
of harm. For example, harm from an overdose of opioid medications could be detected
through review of the use of naloxone. This post hoc logic is consistent through most
triggers, including “rescue drugs” or abnormal laboratory values. However, our study's
method was designed to detect near-miss prescribing errors that were corrected before
reaching patients.
