Keywords adverse drug events - natural language processing - clinical decision support systems
Background and Significance
Background and Significance
Adverse drug reactions (ADRs) comprise a serious health care problem, causing substantial
morbidity and mortality.[1 ]
[2 ]
[3 ] They generate preventable emergency department visits and hospital admissions, prolong
hospital stays, and increase health care costs.[1 ]
[2 ]
[3 ]
[4 ]
[5 ]
[6 ]
[7 ] Providers often fail to recognize ADRs quickly enough or at all.[1 ]
[2 ] As new drugs become available and patients take more medications, providers will
face increasing challenges in recognizing their patients' ADRs. This article describes
the development and evaluation of the Adverse Drug Effect Recognizer (ADER). Authors
hypothesized that ADER could help physicians recognize and address patients' ongoing
preadmission ADRs.
Forty years ago, the World Health Organization[8 ] (WHO) defined ADR as “a response to a drug that is noxious and unintended and occurs
at doses normally used in man for the prophylaxis, diagnosis or therapy of disease,
or for modification of physiological function.” Edwards and Aronson[9 ] later defined an ADR as “an appreciably harmful or unpleasant reaction, … related
to the use of a medicinal product, which predicts hazard from future administration
and warrants prevention or specific treatment, or alteration of the dosage regimen,
or withdrawal of the product.” They explained that adverse drug effect and ADR refer to the same concept, but while ADR is from the point of view of the patient, adverse drug effect is from the point of view of the drug.[9 ] The ADER system identifies electronic health record (EHR)-based patient findings
that match known adverse effects of patients' preadmission medications. Only the patient's
physician can determine whether matched findings actually represent true ADRs, since
a patient's underlying disorder might alternatively have caused the finding. For clarity,
the authors use the term adverse effect for known medication side effects. Authors use ADR for clinician-confirmed true ADRs. Correspondingly, authors refer to findings mentioned
in the EHR that match known adverse effects as potential ADRs .
A 2008 systematic review of 25 studies found that, on average, 5% of hospital admissions were associated with ADRs.[5 ] A 2014 retrospective analysis of 7 million U.S. hospital discharge abstracts similarly
found a combined prevalence of 5.6% for preadmission and inpatient ADRs.[10 ] The Centers for Disease Control and Prevention (CDC) noted that 82% of American
adults take at least one medication, and 29% take five or more.[2 ] A 2014 study found those taking three or more medications were at higher risk for
ADRs, with an increased risk for those taking more medications.[11 ] Additionally, the CDC estimated that ADRs result in more than one million emergency
department visits and 280,00 hospitalizations in the United States annually.[1 ]
[2 ] Thus, ADRs generate substantial health care costs.[12 ]
[13 ]
[14 ]
[15 ]
[16 ] In 2006, the Institute of Medicine estimated that the U.S. spent $3.5 billion on
extra medical costs due to ADRs, and that 40% of the costs of outpatient ADRs were
preventable.[2 ]
Over the past three decades, investigators have developed automated approaches to
ADR detection and prevention. Early systems identified ADRs that clinicians already
had recognized (and presumably addressed). Those systems surveyed inpatient records
for sudden discontinuations of medications, administrations of known antidotes, and
billing codes indicating ADRs.[17 ]
[18 ] In the 1990s, University of Utah researchers implemented such a system; it detected
significantly more verified ADRs than traditional reporting methods.[19 ]
[20 ]
[21 ] Bates et al later showed that automated ADR detection using diagnosis codes, allergy
rules, and text analysis were effective, although with lower predictive values than
intensive retrospective manual chart reviews.[22 ]
[23 ]
[24 ] Nevertheless, they also found that combining discharge summaries, laboratory results,
and medication records could identify valuable ADR-related information.[25 ]
[26 ]
[27 ] Their text-searching tools revealed more potential ADRs than simply reviewing laboratory
results or diagnostic codes.[23 ]
Since 2000, improved natural language processing (NLP) techniques have enabled researchers
to better identify ADRs from clinical documents. For example, researchers from Columbia
University applied NLP to discharge summaries and the biomedical literature to retrospectively
detect potentially unrecognized ADRs and other disease–drug associations.[28 ]
[29 ]
[30 ]
[31 ]
[32 ] Researchers at Stanford University also utilized NLP to extract ADR information
for pharmacovigilance studies.[33 ]
[34 ]
[35 ]
[36 ]
[37 ] The latter studies demonstrated the need to apply additional drug knowledge to distinguish
ADRs from confounders such as indications for drug therapy.
Past work utilizing NLP in combination with data on known adverse effects forms the
conceptual basis for the current ADER study. Previous studies[38 ]
[39 ] have validated NLP and text-mining methods for retrospectively identifying adverse
effects from EHR notes. On admission, ADER automatically extracts inpatients' medications,
symptoms, and previous diagnoses using NLP of provider-generated admission history
and physical examination (H&P) notes. The system then cross-references these data
against a set of known adverse effects and generates patient-specific alerts in the
EHR. This article describes an intervention study to determine whether ADER could
improve providers' recognition of ADRs. The current study is one of the first to evaluate
a system that uses NLP to warn clinicians of potential ongoing ADRs at the time of
hospital admission.
Objective
The goal of this project was development of the ADER system and preliminary evaluation
of whether it could improve inpatient providers' recognition of ongoing ADRs due to
outpatient medications. Authors hypothesized that physicians receiving ADER alerts
would hold or discontinue suspected ADR-causing medications at a greater rate than
a retrospective control group of similar physicians who did not receive alerts. The
study assessed ADER's effect on ADR recognition by comparing admission medications
to inpatient and discharge medication orders for patients in both the intervention
and control groups.
Methods
The ADER system utilizes several previously validated and publicly available NLP tools.
SecTag[40 ] identifies section headers in H&P notes; the KnowledgeMap Concept Identifier (KMCI)[41 ]
[42 ] extracts clinical concepts from text in a manner similar to MetaMap,[43 ] and MedEx[44 ] extracts medication names and dosage information from clinical texts. MedEx was
originally trained on data similar to that used for the ADER evaluation.[44 ] A previous study confirmed that H&P notes at the authors' institution accurately
capture patients' preadmission medications.[45 ] In the current study, KMCI and ADER used the 2013ab version of the Unified Medical
Language System (UMLS); ADER and MedEx utilized the 3 February 2014 release of RxNorm.[46 ]
[47 ]
ADER Design
Upon each patient's hospital admission, clinicians perform a thorough evaluation recorded
in the H&P notes. During the current study, immediately upon generation of a new H&P
in the hospital EHR, ADER copied the note to its database for processing. First, ADER
ran SecTag to parse sections of the note, such as Chief Complaint , History of Present Illness , Past Medical History , Medications , and Family Medical History . Next, ADER used KMCI to identify the patient's symptoms, findings, and previous
diagnoses (diseases) mentioned in each section and mapped them to the Systematized
Nomenclature of Medicine – Clinical Terms (SNOMED-CT) clinical vocabulary via UMLS.[48 ] Using an implementation of the NegEx algorithm,[49 ] KMCI identified whether any identified concepts were characterized as absent (e.g.,
“no fever,” “patient denied cough,” etc.). The ADER system ignores such negated items
and also terms mentioned in sections unlikely to represent the patient's own findings
(e.g., Family History as identified by SecTag). Additionally, ADER used regular expressions to extract
numerical values from the H&P Vital Signs section, mapping abnormal findings to SNOMED-CT (e.g., hypotension for systolic blood pressure < 90). Similarly, ADER mapped abnormal admission of EHR
laboratory results to SNOMED-CT concepts (e.g., hypokalemia for serum potassium < 3.2 mEq/L). Next, ADER utilized MedEx to identify the patient's
current medications from the H&P's Medications section. It automatically mapped drugs to their generic ingredients and represents
them using RxNorm.[47 ]
Finally, ADER cross-referenced the patient's recorded medications and clinical findings
against a database of known adverse effects. Thus, ADER identified potential ongoing
ADRs due to the patient's preadmission (outpatient) medications. The ADER adverse
effect database derives from the authors' previous work on the Drug Evidence Base
(DEB2), a knowledge base containing drug indications and adverse effects extracted
from five public domain sources using automated methods.[50 ] See details in the “ADER Evaluation” section below.
A patient's underlying medical conditions can also potentially explain findings that
ADER labels as potential ADRs. For example, a patient taking Lisinopril might have
a cough due to chronic obstructive pulmonary disease, and not due to an ADR. Prior
to the current study, the authors calculated the frequency of mentions of each potential
ADR finding in a corpus of 350,000 historical H&P notes. Authors then discerned how
often such mentions occurred in patients taking potentially ADR-causing medications,
as well as how often such mentions were associated with patients' various discharge
diagnoses when no ADR-causing medication was recorded. Based on those calculated relative
risk scores, ADER determined whether a patient's current diagnoses plausibly might
have caused a finding that ADER identified as a potential ADR. The system warned clinicians
of such confounding diseases in its ADR alerts. [Fig. 1 ] provides an overview of ADER workflow.
Fig. 1 Complete Adverse Drug Effect Recognizer (ADER) workflow diagram. HPI, history of
present illness; PE, physical exam; ROS, review of systems.
Within a few minutes of a clinician creating an H&P note, ADER identified any potential
ADRs within the note and generated patient-specific alerts. Each alert displayed the
potential ADRs, suspected causative medications, specific evidence extracted from
the chart, and any possible confounding diseases. [Fig. 2 ] shows sample ADER alerts. Physicians could review the alerts in multiple places
in Vanderbilt's EHR, “StarPanel”:[51 ] the EHR Team Summary (used by all care providers); in StarPanel EHR progress note
template forms (used to create clinicians' daily progress notes); and, in frequently
used, patient-specific, paper-based reports (used by physicians for daily rounds).
As part of the current study, alerts included a brief embedded questionnaire to collect
feedback from intervention providers.
Fig. 2 Several Adverse Drug Effect Recognizer (ADER) alerts from the study, including potential
false positive alerts (renal insufficiency, cough, and edema are potential ADRs, but
could also be caused by the patient's kidney failure, chronic obstructive pulmonary
disease [COPD], and heart failure).
To prevent alert fatigue,[52 ] and also at the request of physicians contributing to study design, ADER used noninterruptive
alert delivery mechanisms (i.e., its passive displays did not break EHR workflows).
The system automatically removed alerts from the EHR when: (1) a provider responded
to the alert questionnaire; (2) the patient was discharged; or (3) alert-related questionnaire
went unanswered for 5 days.
ADER Evaluation
The ADER intervention study tested the hypothesis that ADER could help providers to
better recognize and respond to inpatients' preadmission ongoing ADRs. The study focused
on one common class of medications, antihypertensives, which have a myriad of known
side effects. The complete set of antihypertensive adverse effects detectable by ADER
appears in the Supplementary Material ([Supplementary Tables S1 ]
[S2 ]
[S3 ], available in the online version). A team of clinical pharmacists reviewed the antihypertensive
adverse effects from the previously mentioned DEB2 database, making any necessary
clinically relevant revisions prior to initiation of the study. After the target adverse
effects had been determined, the authors utilized the aforementioned corpus of 350,000
previous-to-current-study H&Ps to refine KMCI's recognition of those specific effects.
This process captured common misspellings, abbreviations, and lexical variants, which
were used to improve KMCI recognition of them. To determine the accuracy of concepts
recognized by ADER's NLP during the study, the authors analyzed a sample of 100 H&P
notes where alerts had been issued. This manual audit attempted to verify that all
ADER-identified clinical findings and medications mentioned in alerts were actually
present in patient's records.
The Vanderbilt University Institutional Review Board (IRB) approved the study (IRB
#141341). Indirectly, the study involved participating house officers' attending physicians
and adult inpatients; they did not require informed consent because the intervention
simply informed house officers of potential ADRs. The system did not directly change
patients' medications. All treatment and medication decisions remained entirely with
the care team.
The ADER intervention study began on August 1 and ended on October 31, 2015. The intervention
group included consented interns and residents on the General Internal Medicine wards
of the Vanderbilt University Hospital. The study length was based on pragmatic convenience—authors
were uncertain prior to the study whether the number of alerts that ADER generated
would cause alert fatigue. Therefore, consented participating residents and interns,
as well as the Director of the Internal Medicine Residency program, agreed to a 3-month
trial period as the maximum tolerable duration (in the event that warnings became
onerous). The retrospective control group consisted of Internal Medicine interns and
residents assigned to the same wards as intervention physicians during a 3-month period
just prior to the activation of the system (April 1 through June 30, 2015). To enable
comparison of intervention physicians' behaviors to that of controls, the authors
ran ADER on the retrospective control group's previous H&P notes, identifying instances
when alerts would have been issued had the system been active at the time.
For both intervention and control groups, the study analyzed all clinicians' prescribing
behaviors using NLP-based analysis of admission notes, inpatient medication orders,
and discharge summaries. This process determined whether preadmission medications
from the H&P were continued, held or discontinued during the hospital stay and at
discharge. The study compared medication hold/discontinue rates of all intervention
clinicians who received alerts and all control physicians who would have received
alerts had the system been active. The study also determined separately whether the
subgroup of intervention providers who completed the alert-embedded survey (and therefore
definitely had seen the alerts) were more likely to change suspected ADR-causing medications
compared with all nonalerted control group providers. When evaluating discharge medications,
the study only analyzed notes for those patients both admitted and discharged by study
subjects (i.e., Internal Medicine house staff). Patients transferred to the care of
physicians on another service (e.g., Surgery) were not included in analysis of discharge
medications.
When comparing preadmission medications to inpatient medication records , the authors evaluated the difference between intervention and control groups for
suspected ADR-causing medications (1) ordered ever during the inpatient stay, (2)
ordered in the first 24 hours of the stay, (3) ordered, then stopped or held, as well
as (4) suspected ADR-causing medications stopped or held after intervention providers
responded to the alert survey versus the overall hold rate in control group. When
comparing preadmission medications to discharge summaries , the authors evaluated the difference between intervention and control groups for
suspected ADR-causing medications (5) held at discharge and (6) held at discharge
for intervention providers who responded to the alert survey. The primary outcome
was (1), the difference between groups for suspected ADR-causing medications ordered
ever during the inpatient stay.
Whenever intervention subjects' survey responses indicated that they would make or
had made changes to suspected ADR-causing medications, the study identified (by analyzing
actual orders issued) whether the provider actually followed-through with the medication
order changes. An original long version of the survey questionnaire had a low response
rate (12%); subjects reported that it was cumbersome and had too many questions. Therefore,
a shorter survey ([Fig. 2 ]) replaced the longer version approximately 3 weeks into the study. Only survey responses
obtained during the 9 weeks using the shorter-form survey appear in the results. The
authors reviewed open-ended survey comments submitted by intervention subjects and
classified them based upon similar themes. Additionally, the authors met with study
subjects, including the Chief Resident and Director of the Internal Medicine Residency
Program, to discuss their experiences and suggestions at the completion of the intervention.
Statistical Analysis
The study characterized categorical variables using frequencies and proportions and
used medians and interquartile range (IQR) to describe continuous variables. To determine
whether ADER alerts affected providers' behaviors, authors analyzed medication order
changes during the inpatient stay and at discharge. The analysis compared the rates
of medication changes between the control and intervention groups using chi-square
tests with a significance level of 0.05. When performing secondary analyses on data,
the authors used Bonferroni correction to adjust the significance level for all comparisons
to account for multiple testing. All statistical analyses used R version 2.15.2.
Results
[Table 1 ] illustrates ADER study results for both the intervention and retrospective control
groups. The 3-month preintervention control group included 2,666 H&P notes written
by 138 interns and residents. During the 3-month intervention period, the ADER scanned
2,312 H&P notes written by 137 interns and residents. No significant differences occurred
between the intervention and control groups for any metrics shown in [Table 1 ] (p > 0.05). The ADER system detected potential ADRs in 30 to 32% of both groups' notes,
with 78 to 79% triggered by H&P text findings and 21 to 22% by laboratory results.
The median time from when an intervention group physician saved an admission note
to the EHR to when ADER posted an alert was 8 minutes (IQR = 6–9 minutes). The median
time between ADER issuing an alert and an intervention subject responding to the embedded
survey questions was 24 hours (IQR = 17–37 hours).
Table 1
ADER study results for intervention and control groups
H&Ps
Unique patients
Notes with potential ADRs
Average number of antihypertensive medications per note
N
N
%
N
%
Notes without potential ADRs
Notes with potential ADRs
Intervention
2,312
2,049
89
701
30
0.73
2.4
Control
2,666
2,352
88
867
32
0.76
2.4
Potential ADRs per alert
Adverse effect source
Number of potential ADRs found per alert
H&P text
Labs
Min.
25%
Median
75%
Max
Intervention
2.3
79
21
1
1
2
3
15
Control
2.4
78
22
1
1
2
3
23
Admissions with discharge note
Unique patients with discharge note
Average length of stay
Number of providers
N
%
N
%
Total
With discharge
Intervention
1,835
79
1,646
90
115 h
137
137
Control
2,160
81
1,916
89
112 h
138
128
Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction; H&P,
history and physical examination.
ADER NLP Accuracy
Manual expert review of 100 intervention group H&P notes for which ADER had issued
potential ADR alerts revealed that 98% of the 157 medications that ADER identified
as current therapies were correct. In three instances, ADER incorrectly identified
drugs that the H&Ps described as previously discontinued. Of 136 total adverse effects
(findings or diseases) that ADER identified from H&P notes, 116 (86%) were confirmed
as “present” in the chart. Of the 20 incorrectly identified adverse effects, 15 were
mentioned as being absent but were not recognized as “negated” by KMCI due to use
of nonstandard wording. The remaining misidentified concepts were mentioned only as
possible points of differential diagnosis or past inactive diagnoses. Of the 30 clinical
findings the ADER recognized from laboratory results, 29 (97%) were accurate. The
only incorrectly classified numerical laboratory result was “hyperkalemia” that had
been noted in comments to be due to a hemolyzed sample.
Comparing Admission Medications to Inpatient Medication Orders
The intervention group ordered significantly fewer suspected ADR-causing preadmission
medications at any point during the inpatient stay compared with the control group
(47% vs. 58%, p < 0.001). Similarly, during the first 24 hours after admission, intervention physicians
ordered only 28% of 1,140 suspected ADR-causing preadmission medications flagged by
ADER; the control group had ordered 39% of 1,468 suspected ADR-causing preadmission
medications during the same interval (28% vs. 39%, p < 0.001). See [Table 2 ].
Table 2
Preadmission medications compared with ordered inpatient medications, testing the
null hypothesis that the proportion for the intervention group was equal to the proportion
for the control group
Suspected ADR-causing medications
Ordered ever during inpatient stay
Ordered in first 24 hours
Ordered, then stopped or held
N
Proportion
N
Proportion
N
Proportion
Intervention
1,140
532
0.467
315
0.276
91
0.171
Control
1,468
846
0.576
570
0.39
125
0.148
Difference
(95% CI)
p -value
–0.109
(–0.149, –0.070)
p < 0.001
–0.114
(–0.149, –0.075)
p < 0.001
0.023
(–0.018, 0.065)
p = 0.28
Abbreviations: ADR, adverse drug reaction; CI, confidence interval.
Among suspected ADR-causing medications ordered during the inpatient stay, 84% (711/846)
were ordered within 24 hours of admission in the control group; in the intervention
group, 77% (410/532) were ordered within 24 hours of admission (p = 0.002). Suspected ADR-causing medications were stopped before patient discharge
at rates of 14 and 16% for the control and intervention groups, respectively (p = 0.40). Among medications started after 24 hours into the inpatient stay, subsequent
hold rates were 19 and 20% in the control and intervention groups, respectively (p = 1.00).
Among suspected ADR-causing preadmission medications ordered at any point in the inpatient stay, the rate at which they were subsequently held or discontinued
did not vary significantly between groups (see [Table 2 ], p = 0.28). However, secondary analysis, as shown in [Table 3 ], revealed that the subset of intervention providers who responded to alert-embedded
surveys—and therefore who definitely viewed the alerts—subsequently held or discontinued
25% (41/167) of the then-active suspected ADR-causing preadmission medications. This
rate is significantly different than the aforementioned control group rate (25% vs.
15%, p = 0.003). In the intervention group, the median time between responding to the survey
and subsequent medication changes was 25 hours (IQR = 13–66).
Table 3
Preadmission medications ordering during inpatient stay later stopped or held, testing
the null hypothesis that the proportion for the intervention group after survey response was equal to the overall proportion for the control group
Suspected ADR-causing meds
N
N
Proportion
Intervention
Active order at time of ADER alert survey response
167
Stopped or held after survey response
41
0.246
Control
Ordered ever during the inpatient stay
846
Ordered ever, then stopped or held
125
0.148
Difference
(95% CI)
p -value
0.098
(0.025, 0.171)
p = 0.003
Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction; CI,
confidence interval.
Comparing Admission Medications to Discharge Medications
Approximately 20% of intervention and control group discharge summaries were not generated
by study subjects and thus were excluded from review. [Table 4 ] shows hold/discontinuation rates of suspected ADR-causing preadmission medications
at discharge for the remaining 80% of discharge summaries.
Table 4
Preadmission medications compared with discharge medications testing the null hypothesis
that the proportion for the intervention group was equal to the proportion for the
control group
Suspected ADR-causing preadmission medications: full intervention group versus full
control group
Suspected ADR-causing preadmission medications; intervention group survey respondents versus full control group
Total
Medication missing from discharge note
Held or discontinued at discharge (explicitly)
Total
Held or discontinued at discharge (explicitly)
N
N
Proportion
N
Proportion
N
N
Proportion
Intervention
927
104
0.112
184
0.198
258
97
0.376
Control
1,225
75
0.061
287
0.234
1,225
287
0.234
Difference
(95% CI)
p -value
0.051 (0.025, 0.076)
p < 0.001
–0.036 (–0.072, 0.0001)
p = 0.053
0.142
(0.076, 0.208)
p < 0.001
Abbreviations: ADR, adverse drug reaction; CI, confidence interval.
When discharge notes did not mention previously alerted preadmission medications,
the study could not definitively determine whether the medications had been held or
stopped at discharge (104/927 = 11% from intervention, 75/1,225 = 6.1% from control,
p < 0.001). Actual specific mentions in discharge notes of holds/discontinuations for
alerted medications did not differ significantly: 20% (184/927) in the intervention
group and 23% (287/1225) in the control group (p = 0.053). However, intervention providers who responded to the alert-embedded survey
held or discontinued 38% (97/258) of suspected ADR-causing medications at discharge,
significantly different (p < 0.001) than the overall 23% rate for the control group.
Survey Questionnaire Results and Provider Comments
Intervention subjects responded to 53% (298/560) of the short-version alert-embedded
questionnaires (see [Table 5 ]). For survey respondents who stated that they would change the patient's therapy
due to identified ADRs, subsequent medication holds or dosage decreases occurred more
than 90% of the time. For the 38 intervention subjects who responded that they were
uncertain whether they would make a change, analysis identified 24 instances (63%)
where one or more suspected ADR-causing medications was subsequently changed. Nearly
three-quarters of survey responses indicated alerts either somewhat or fully merited
consideration for their patients. More than half said alerts were helpful or possibly
helpful in managing the patient.
Table 5
Responses to ADER alert survey questionnaires (n = 298)
Question
Response
N
(%)
N
(%)
N
(%)
Did any of these alerts merit at least passing consideration for this patient?
Yes
144
(48)
Somewhat
74
(24)
No
80
(26)
Were any of these alerts helpful in managing this patient?
Yes
87
(29)
Possibly
80
(26)
No
131
(43)
Will you (or did you already) change patient's therapy due to these possible ADRs?
Yes
91
(30)
Uncertain
38
(12)
No
169
(56)
Abbreviations: ADER, Adverse Drug Effect Recognizer; ADR, adverse drug reaction.
Approximately 30% of the 52 optional free-text survey comments indicated that ADER
was correct in identifying an ADR. Conversely, 30% of the survey comments stated that
claimed adverse effects were more likely due the patient's underlying medical conditions
(usually those recognized by ADER as possible confounders). For example, when ADER
identified renal insufficiency or acute kidney injury as an ADR, providers often indicated that the patient's existing end-stage renal disease was the more likely cause of impaired renal function.
In ∼20% of survey comments, providers indicated that the medications were held, but
not due to the ADER-recognized ADRs. In at least two cases, however, the H&P Plan section indicated that a provider had in fact held the medications for the reasons
identified by ADER. In 10% of the comments, providers indicated that the patients
were not suffering from the suspected ADR. In one of those cases, ADER had recognized
“gout” as a possible ADR. The patient had a previous diagnosis of gout, but no acute
gout exacerbation was present on admission. In another case, ADER “incorrectly” identified
that the patient had a cough because of the phrase, “Hurts with movement, cough, sneezing,
position changes.” Several survey respondents suggested that they had already recognized
the ADR suggested by ADER.
Study subjects also provided feedback to the authors directly. Participating providers
generally viewed ADER favorably, but suggested that ADER might be more useful for
those medications that internists prescribe less frequently than antihypertensives,
such as psychiatric medications. They also expressed that potential confounders included
in ADER alerts were helpful in assessing the veracity of suspected ADRs and in considering
whether medications were possibly exacerbating the recognized findings.
Medication Holds Analyzed by ADR
Among the 940 intervention group alerts, 41 were distinct adverse effects (patient
findings/disorders). The most frequent alert-related conditions (occurring more than
50 times) were: renal insufficiency, light headedness, edema, dizziness, hypotension,
and syncope. Suspected causative medications were held at discharge at rates of 35%
for hypotension, 28% for renal insufficiency, 42% for hyperkalemia, 36% for hypokalemia,
and 50% for hyponatremia. The Supplementary Material ([Supplementary Tables S1 ]
[S2 ]
[S3 ], available in the online version) provides hold/discontinue rates for all medications
with detected ADRs. The online Supplementary Material also reports comparisons of
hold rates between suspected ADR-causing medications and those not suspected of causing
ADRs.
Discussion
The current study illustrates the potential for automated alerting systems like ADER
to help providers identify unrecognized ADRs. Physicians receiving ADER alerts held
or discontinued more suspected ADR-causing medications at multiple points in the patient
care workflow compared with physicians who did not receive alerts. In survey responses,
physicians stated that ADER alerts more than half the time helped with patient management.
Analysis of ADER's accuracy revealed that the majority EHR-extracted findings and
medications were correct, but also revealed opportunities for improvements in NLP,
particularly in regards to negation detection.
Intervention providers receiving ADER alerts ordered significantly fewer suspected
ADR-causing preadmission medications in the first 24 hours after admission and at
any time during the inpatient stay. As a whole, intervention providers receiving alerts
were no more likely to discontinue suspected ADR-causing preadmission medications
at discharge than the control group (although they had ordered fewer such medications
overall at admission). Nevertheless, providers who responded to the alert-embedded
survey were more likely to hold or discontinue ADR-causing medications during the patient's hospitalization
and at discharge. Two explanations are possible—either providers who believed that
a potential ADR alert was valid were more likely to respond to the survey, or providers
who believed ADER alerts were true more often addressed potential ADR-causing medications
at discharge. Further study could determine whether ADER alerts would be more effective
if they interrupted workflows, since the current study used noninterruptive alerts.
Why intervention physicians in some instances ordered previously alerted ADR-causing
medications later during an admission is also a topic for further research. A possible
explanation is that when ADR-related findings resolved, re-starting the medication
was appropriate (e.g., hypotension in patients with baseline hypertension).
From a technical standpoint, the ADER approach is generalizable to many inpatient
facilities. The system only requires that a hospital have an EHR system that can electronically
export (under appropriate security) admission H&P notes and laboratory values as they
are generated to a generic desktop-class computer running the ADER software. Another
key requirement is the hospital EHR's ability to incorporate ADER-generated alerts
into the EHR workflow in locations likely to be viewed by clinicians.
Comparison with Prior Research
Use of NLP to identify potential ongoing ADRs from outpatient medications at the time
of admission is novel. A substantial body of work exists regarding other methods of
ADR detection and associated determinations of physicians' responses. In 2017, Dexheimer
et al reported that providers in a pediatric hospital, after receiving inpatient dosing,
allergy, and drug–drug interaction alerts cancelled or modified medication orders
an average of 12.6, 7.0, and 16.6%, respectively.[53 ] A 2003 study of five adult primary care practices found that physicians accepted
only 8.8% of drug allergy and 10.6% of high-severity drug–drug interaction alerts.[54 ] Further analysis revealed physicians were more likely to override alerts for renewals
than new prescriptions and that house officers were less likely to prescribe alerted
medications than other physicians. A different study suggested that house staff were
more likely to override ADR alerts than staff physicians.[55 ] In a classic 1976 study, McDonald evaluated provider responses to electronic reminders,
including suggestions to observe a physical finding or ask about a specific symptom,
order a diagnostic study or change or start a therapeutic regimen.[56 ] The study found physicians responded to 51% of events when given electronic reminders
compared with 22% of events when they did not receive reminders. Of note, few such
reminders involved medication changes due to potential ADRs. The study indicated that
physicians comply with a higher proportion of electronic recommendations when protocols
are more precise.
The current study's physician alert response rates exceed the aforementioned previous
studies regarding other types of ADR alerts. A possible explanation is that the current
study named in its alerts the suspected ADR symptoms that were present in the patient
at admission. The ADER methodology is significantly different than that of the above-mentioned
efforts, as well as the adverse effect detection studies discussed in the background.
The previous efforts utilized sudden discontinuations of medications, administrations
of known antidotes, and ADR-related billing codes, all of which indicate that providers
had already recognized the ADRs. They also warned about dosing errors, allergies,
and possible drug–drug interactions, which often do not take into account the actual
symptoms being experienced by the patient. The ADER system is novel in that it utilizes
NLP of provider-generated admission notes to warn clinicians about potential ongoing
ADRs, as evidenced by documented symptoms and findings, due to outpatient medications.
It has the potential to detect unrecognized ADRs and provide a safeguard to ensure
all providers are aware of the ADRs their patients may be experiencing.
Limitations and Future Work
Several limitations pertain to the study. The intervention only included Internal
Medicine interns and residents as alert recipients. That survey responses followed
alerts by ∼24 hours suggests that house officers may have waited to discuss alerts
with the ward team and attending physicians before acting. Additionally, house staff
often rotate “off service” so that the physicians writing discharge orders may not
have been the ones who received the initial ADER alerts at the time of admission,
even though the discharging physician qualified as study participants by membership
in the Internal Medicine house staff. A wider range of subjects (e.g., academic attending
physicians, community-based physicians, nurse practitioners, physician assistants)
should be evaluated in teaching and nonteaching settings. The study limited the ADER
intervention to antihypertensive medications so that potential alert fatigue would
not confound initial evaluation of ADER's utility—as might occur if all categories
of medications generated alerts. The study targeted the commonly prescribed antihypertensive
class because they cause a wide variety of ADRs. Admittedly, providers who regularly
prescribe these medications are more likely to be aware of their potential ADRs, and
study participants suggested that ADER might be more useful for medications they prescribe
less frequently. In the future, ADER should be expanded, in consultation with clinical
experts, to cover additional drug classes and their related ADRs, with particular
focus on uncommon or severe ADRs.
The study analyzed 3 months of intervention data compared with a similar-duration
retrospective control group. While analysis showed the control group to be comparable
to the intervention group in nearly all respects, the groups contained different participants
and the admissions occurred at somewhat different times in the academic year (controls
were soon-to-be promoted house officers; intervention subjects had arrived in their
current positions 1 month prior to study initiation). A randomized controlled trial
involving multiple institutions and all drug classes could provide a more definitive
result than the current study. Before undertaking such a study, the potentially correctable
flaws in ADER's NLP methods (identified in this study) should be addressed.
To evaluate the potential of this methodology, this study focused only on the accuracy
of the ADER alerts that were presented. Future studies with additional drug classes
should consider both precision and recall to determine which potential ADRs ADER misses
(i.e., fails to generate alerts when appropriate to warn clinicians). To reduce alert
fatigue, more advanced confounder detection should be studied with the goal of filtering
out false-positive alerts when potential ADRs have a significantly more likely clinical
cause. Future studies will also include more in-depth qualitative analysis of providers'
impressions and resultant behavior.
Conclusion
Intervention subjects reduced ordering of suspected ADR-causing medications at multiple
points during the admission compared with the control group. Systems like ADER have
the potential to improve both recognition of adverse effects and discontinuation of
medications causing preadmission ADRs. Future ADER-like systems must supply relevant
information to care providers at the right place and time to improve patient safety,
potentially decrease cost, and improve the quality of care.
Clinical Relevance Statement
Clinical Relevance Statement
This study shows that using NLP extraction of patient preadmission medications and
findings to generate real-time ADR alerts can affect provider ordering behavior. This
approach could be generalized to other facilities that use electronic health records.
Clinical decision support systems like ADER could help to improve the safety of care
delivery in the future.
Multiple Choice Questions
Multiple Choice Questions
Approximately what percentage of ADER-identified adverse drug reactions were found
in the text of the admission note, as compared with solely derived from laboratory
test results?
How often did intervention clinicians receiving an alert order the potentially offending
medication during the hospital stay, and how often did control physicians do so?
