Clinician Perceptions of Timing and Presentation of Drug-Drug Interaction Alerts

Abstract

Objective Alert presentation of clinical decision support recommendations is a common method for providing information; however, many alerts are overridden suggesting presentation design improvements can be made. This study attempts to assess pediatric prescriber information needs for drug–drug interactions (DDIs) alerts and to evaluate the optimal presentation timing and presentation in the medication ordering process.

Methods Six case scenarios presented interactions between medications used in pediatric specialties of general medicine, infectious disease, cardiology, and neurology. Timing varied to include alert interruption at medication selection versus order submission; or was noninterruptive. Interviews were audiotaped, transcribed, and independently analyzed to derive central themes.

Results Fourteen trainee and attending clinicians trained in pediatrics, cardiology, and neurology participated. Coders derived 8 central themes from 929 quotes. Discordance exists between medication prescribing frequency and DDI knowledge; providers may commonly prescribe medications for which they do not recognize DDIs. Providers wanted alerts at medication selection rather than at order signature. Alert presentation themes included standardizing text, providing interaction-specific incidence/risk information, DDI rating scales, consolidating alerts, and providing alternative therapies. Providers want alerts to be actionable, for example, allowing medication discontinuation and color visual cues for essential information. Despite alert volume, participants did not “mind being reminded because there is always the chance that at that particular moment (they) do not remember it” and acknowledged the importance of alerts as “essential in terms of patient safety.”

Conclusion Clinicians unanimously agreed on the importance of receiving DDI alerts to improve patient safety. The perceived alert value can be improved by incorporating clinician preferences for timing and presentation.

Keywords

drug interactions - electronic health records - health personnel alert fatigue - medication alert systems

Background and Significance

Clinical decision support (CDS) forms a critical component of electronic health records (EHRs) by supporting provider's decision making to improve patient care. One common form of medication-related CDS involves identification and alerting of potential drug interactions. These alerts have the potential to reduce adverse drug events and influence provider ordering behavior.[1] [2]

A large proportion of alerts, even those that are clinically significant, are overridden. This suggests that these alerts are not helpful as presented.[3] [4] [5] [6] Effective CDS should follow the five rights of alert presentation, specifically that CDS interventions must deliver the right information, to the right person, in the right format, through the right channel, at the right point in the workflow.[7] Further, studies of alert acceptance have highlighted the need for alignment of alert design with human factors principles.[8] Human factors analysis approaches to research have been recommended as helpful in understanding the cognitive processes in decision making involving computerized systems.[9] Researchers have described these concepts arguing that alert design is important for information assimilation and a greater predictor of acceptance by the end user, even more than the content of the alert itself.[10] [11]

Phansalkar et al identified the specific components of alert design that would improve its acceptance using a validated questionnaire, I-MeDeSA, to assess drug–drug interaction (DDI) alerts for compliance with human factor principles.[12] Common weaknesses identified were alert prioritization, concise information regarding consequences of alert override, and clinical management information. Of note, further studies of I-MeDeSA have suggested tool refinement and clarification may be needed to allow for improved use.[13] [14] Another systematic qualitative review of usability flaws of medication alerting functions identified 13 categories of usability flaws including low signal-to-noise ratio, alert content issues, and alert appearance issues including timing of the alert and mode of presentation.[15] Poorly designed alerts contribute to cognitive burden in an already demanding environment and may therefore pose a risk to patient safety.[16] [17] [18]

Pediatricians are a unique group to assess alert acceptance and information needs as children and infants often differ from adults in their metabolism and response to medications.[19] Thus, pediatric clinician information needs may differ from adult practicing clinicians. This study takes place in a free-standing quaternary pediatric hospital with physicians with general medical training in pediatrics and pediatric subspecialties.

Objectives

In this study, we conducted a qualitative evaluation to examine alert design components, specifically the timing of alert presentation. Additionally, alerts were perceived as helpful to pediatric clinicians.

The objectives of this study were to:

1. Assess pediatrician information needs when DDI alerts are presented in the EHR.

2. Evaluate the optimal timing of alert presentation within the medication order entry process to support providers' cognitive decision making and to minimize inefficiencies.

Methods

Settings and Participants

We conducted this study at a quaternary free-standing pediatric hospital in the northeastern United States with Cerner Multum-supported CDS that undergoes monthly maintenance updates. We employed a purposeful sampling strategy to recruit key clinicians across a broad range of end-user perspectives[20] [21]: pediatric trainees (interns, residents, and fellows), pediatric attending physicians, pediatric cardiologists, and pediatric neurologists. These end-users demonstrate clinical expertise in pediatrics with a diverse array of subspecialty training. All participants were actively practicing clinicians with expertise in the treatment of children with responsibilities including routinely placing medication orders for patients during the study period (February–July 2015).

We sent recruitment emails to all pediatric interns and residents, listed as institution employees at the time of recruitment, a total of 160 clinicians. Additionally, we contacted 5 cardiologists and 6 neurologists and asked them to recommend other specialists who might be interested in participating in the study.

This study was approved by the Institutional Review Board of Boston Children's Hospital (IRB #P00015071).

Data Collection

To investigate the optimal timing of presenting DDI alerts, we employed participant sessions that included case simulations, verbal protocol, and a series of semistructured interview questions with targeted key informants.[21] [22] These sessions were audio-recorded, transcribed by a contracted service and analyzed.

A computer test environment for placing medication orders was created. We developed four alert presentation scenarios: no alerts, single DDI alert with multiple medication orders, multiple alerts with multiple medication orders, and DDIs presented without interruptive alerts via PowerPlan (Cerner Corp, Kansas City, Missouri, United States), a predetermined set of grouped orders. Alerts included the following information: severity (major-contraindicated), medication, order details, interaction information, and the reason for alert override ([Fig. 1]). Six computer pediatric case simulations were designed in general medicine, infectious diseases, neurology, and cardiology ([Table 1]). These case simulations were based upon major contraindicated DDI pairings to keep consistent with the current local EHR configuration. Participants were asked to place medication orders in the EHR for each of the cases. Five of the six cases contained scenarios which would generate DDI alerts. We varied the timing of alert presentation to occur at the time of medication selection or after the medication selected at the time of signature/submission. In one scenario, automated alert notification was turned off but could be activated at provider discretion. One of the cases contained a group of orders that a prescriber selects for placement simultaneously rather than entering individual orders serially in the EHR (as alert information can be presented differently when orders are entered concurrently rather than consecutively).

Fig. 1 Example of drug–drug interaction (DDI) alert used in the case scenarios (A) with associated interaction alert information (B).

Table 1 Overview of case scenarios

DDI alert type	Scenario content
No DDIs	Clinical scenario: General medicine - asthma exacerbation DDI medication alert: none Alert timing: not applicable
Single DDI with multiple medication orders	Clinical scenario: Infectious disease - urosepsis DDI medication alert: sertraline-linezolid Alert timing: alert at the time of medication entry
Multiple DDIs with multiple medication orders	Clinical scenario: Neurology - pneumonia in a patient with seizure disorder and medical complexity DDI medication alert: lamotrigine-valproic acid, valproic acid–meropenem Alert timing: alert after medication entry
DDIs present without interruptive alert (use of PowerPlan)	Clinical scenario: General medicine - esophagitis DDI medication alert: Sirolimus-Voriconazole Alert timing: no interruptive alert
Single DDI with multiple medication orders	Clinical scenario: Cardiology - atrial flutter DDI medication alert: clopidogrel-omeprazole Alert timing: alert at the time of order submission
Multiple DDIs with multiple medication orders	Clinical scenario: Neurology - pneumonia in a patient with seizure disorder and medical complexity DDI medication alert: lamotrigine-valproic acid, valproic acid–meropenem Alert timing: alert at the time of order submission

Abbreviation: DDI, drug–drug interaction.

We employed “think-aloud” analysis to prompt the participants to describe their actions, decisions, thoughts, and emotions as they interacted with the medication order system and screen visualizations.[23] [24] The participants were asked to complete the tasks as if in an authentic setting. Interruptions by the observers were kept to a minimum throughout participant engagement to minimize the introduction of suggestions or bias. All sessions lasted approximately 60 minutes with cases presented in the same order for each participant and included audio-recorded interviews using a semistructured interview guide developed and pilot tested prior to use in the study. Interviews were performed by principal investigator (K.H.) who is a pediatrician and coinvestigator (M.J.) who is a PhD with background in employing think-aloud, semistructured interviews in prior research. The final guide contained 13 open-ended questions covering the following domains: (1) reasons for alert override; (2) questions about information needs in decision-making; (3) average monthly frequency of medication ordering; (4) knowledge of drug interaction risks; (5) PowerPlan interaction perceptions; (6) general perceptions of DDI alert timing; and (7) suggestions for improvement of the medication order system. These questions were applied at the end of each case scenario.

As new topics emerged during the interviews, we included these topics in all subsequent interviews. For example, one physician spoke about consulting medication reference guides when faced with an alert. All subsequent interviews contained the question, “In what instances do you refer to medication reference guides?”

Data Analyses

“Think-aloud” and semistructured interview data were analyzed in aggregate using a combination of deductive and inductive methods, relying on preliminary codes derived from the interview guide as well as scanning interview transcripts to uncover emergent codes.[25] [26] Using preliminary coding dictionary, two interview transcripts (14%) were reviewed and coded by the principal investigator (K.H.) and a coinvestigator (M.J.). The reviewers discussed and resolved any coding discrepancies. Emergent codes were defined and added to the coding dictionary.[25] The finalized dictionary was reviewed by a third team member with experience in qualitative research and medical informatics (S.P.). Following this process of initial calibration, the two reviewers (K.H. and M.J.) coded all of the remaining interview transcripts. New comments were compared with previously coded statements to ensure consistency of coding.[27] Coded data were reviewed to identify broad themes and subthemes. Frequent discussions between the investigators ensured agreement about these themes and subthemes and permitted additional characterization of the themes.[28] The data analysis was considered complete when theoretical saturation was achieved, that is, no new themes emerged.[28] The qualitative software, NVivo (version 10, QSR, Doncaster, Australia), was used to support the coding and analysis process.

Results

Characteristics of Study Participants

Fourteen pediatricians were recruited in the study: an intern, residents (n = 2), attending physicians (n = 4), pediatric cardiologists (n = 4), and pediatric neurologists (n = 3).

Knowledge of or familiarity with DDI was not reliably associated with frequency of ordering/prescribing those medications. For example, in Case 1 (Supplement 1), built with albuterol, fluticasone, and montelukast, 82% of the respondents (9 of 11) noted that they understood potential DDIs associated with the medications, despite infrequently prescribing them. In Case 2 (sertraline and linezolid) and Case 4 (sirolimus and voriconazole) the situation was reversed. While frequently prescribing the medications individually, 80% (8 of 10) and 60% (3 of 5) of respondents, respectively, indicated a low level of comfort with understanding the effects of these DDIs.

Benefits of Receiving Major Contraindicated Alerts—Meeting Informational Needs of Clinicians

Most participants responded affirmatively when asked whether they would like to receive alerts while placing medication orders. The participants valued alerts for meeting their informational needs ([Table 2]).

Table 2 Key benefits and recommendations for DDI alert presentation

Benefits of alerts	Provide important information Increased awareness to risk of medication interactions Creates a “pause” in the clinical workflow
Alert timing	Provide alerts as early as possible in the order entry process
Alert content	Severity of alert interaction Type of interaction Risk of interaction occurring Alternative medications Links to appropriate clinical resources (formulary, references)
Alert text	Concise description of interaction Recommendations for tests or studies to perform due to DDI Links to pertinent laboratory tests or studies

Abbreviation: DDI, drug–drug interaction.

Participants spoke of the value alerts have in terms of providing reminders of potential DDIs, their severity, opportunities to focus during a busy workflow, and information about alternative medications. A few examples of illustrative quotes are provided below and in [Table 3].

Reminders of potential DDIs, even for familiar medications: “(I) don't mind being reminded… because there's always the chance that at that particular moment I don't remember it or I don't think about it, especially if I'm putting a long list of medications” (Attending). “I don't think it's a bad idea to be reminded of the interactions, even in drugs that you're familiar with because [of] the rate of complications” (Cardiologist).

Reminders about severity of DDIs: “…it is also instructive to say at one time, like not only are these the individual interactions, but like you've got to really be serious that you want to order this medication because it interacts with all four of these things” (Attending).

Opportunities to focus better: “… [I] might be moving too fast” [and the alert] “just pauses” [me] (Neurologist). “…(an) interaction [alert] is somewhat helpful to redirect what I'm doing sometimes”… “it allows me to redirect and think about those interactions of that medication in particular and look it up instead of having 15 things on one screen” (Attending).

Information on alternative medications: “It was helpful to have a suggestion of an alternative medication to use” (Attending).

Opportunities for Improvement

Participants mentioned opportunities for improvement of the alert system were related to alert timing and design, unmet informational needs, and reliance on additional sources of information. Illustrative quotes are provided below, with further quotes included in [Table 4].

Table 3 Themes and illustrative quotes from study participants about benefits of receiving alerts

Themes

Quotes

Benefits of receiving alerts – meeting informational needs

“I do think it would be good to have an alert for the major interactions. I don't mind being reminded of those. I think it's good because I might be too… moving too fast or… Even if I know not to read it, heaven forbid, and it reminds me to make sure. It just pauses me.” (Neurologist 2) “We're typically more concerned with dose related single agent effects than interactions, and I don't think it's a bad idea to be reminded of the interactions.” (Cardiologist 3) “…the attending is vaguely aware of, maybe he saw it [interaction alert] once in fellowship, but like haven't seen it in a while. They know it's a theoretical risk, but kind of the back of their heads are aware of it, and will just continue on with their plan of care. This at least makes me think about it and learn about it.” (Cardiologist 2) “I like the alert to say hey, stop, essentially, and go back and look it up and then figure out why there's an alert and if I should continue or I should not.” (Attending 4) “I think [I like seeing the alert]. Because, you know, I'm new at this, too, so like having some of the, like I probably wouldn't even be aware of those interactions.” (Cardiologist 2) “I think that it [alert] gave a good idea of how severe this interaction was, which would be the main thing I would also want to know.” (Intern) “I think major contraindicated is a fairly… I mean for a clinician, that is a fair red flag.” (Cardiologist 3) “[Alert] was helpful. It recommended alternatives.” (Neurologist 2) “… it's all that risk-benefit stuff, but if you feel like he needs it [medication], then talking about, it was helpful to have a suggestion of an alternative medication to use or an H2 receptor antagonist.” (Attending 3)

Table 4 Themes and illustrative quotes from study participants about opportunities for improvement in alerts

Themes	Quotes
Alert timing and design	“I think I preferred for it [alert] to pop up when I was entering the medications, because then I did not get too far in the ordering process…” (Intern 1) “I liked the first method where it [alert] pops up right away because… especially if it's a matter of just having a dose or something like that. It'd be nice to know that going into writing the order sentence so that you can make that adjustment… Otherwise you have to cancel and rewrite the whole thing. See, I definitely like the order earlier.” (Resident 2) “…if I put everything else and it wasn't until I went to sign that it [alert] popped up, I feel like appropriately or inappropriately my only option would be to cancel everything I had done and start over, which would drive me nuts” (Attending 2) “I don't know if there would be a way to kind of make more of a table or kind of delineate those two things [two interacting drugs].So what's the interaction, why, alternative medications that are recommended, and just for easy visual understanding” (Attending 2) “I think a better idea is to make the alerts more well designed so that they're more informative, they're quicker, and in particular they're more unambiguous to deal with… Don't give people six or seven different choices for things, just choose two or three that really reflect what people are thinking.” (Cardiologist 3) “I think it did require me to read it carefully because at first it said, well, the fact that the alert box popped up is unusual enough that it makes me want to read it. But you do have to go through a few lines to get to the point.” (Attending 3) “I think it's [alert box] a little too dense honestly. It's almost like I wish the two bullet… main bullet points were separated so it was a little bit easier to read because I… I'm trying to get as much information as I can quickly to be list should I order this or should I not. And I find it a little dense to read. I'm not going to lie.” (Attending 4)
Unmet informational needs	“It would have been helpful if there was an explanation of what the risk is… SSRIs are contraindicated with this type of thing because if the risk of serotonin syndrome or kidney failure, and then I could sort of know what I was thinking or maybe that would trigger something.” (Attending 1) “If there'd been a prior EKG available, [that] would be useful.” (Attending 2) “I think if there was a quick click that I could review that [recent EKG or QTc], yeah, that would be helpful.” (Cardiologist 1) “…with the methadone I feel like, well, usually we would say prescribe it anyway, just check an EKG” (Attending 2) “…if that [alert window] would actually give me the nature of the interaction, because, you know… if it was like a minor interaction, that I'd want to be aware of but wouldn't necessarily, you know, make me say, ‘Oh. I’m not going to order this until I get more information'” (Neurologist 3). “want to know more about why he [patient] was on the valproic acid” and was planning “to see if they got levels before he was admitted” (Cardiologist 2)
Additional sources of information
Pharmacists as source of information	“I'd want to call pharmacy first and say I'm getting this alert, does… should I do that... this for this child in this setting, and then I would probably do that” (Resident 2) “It's telling me to consider modification of my order by decreasing the dose of lamotrigine. So I'm going to speak with pharmacy with regard to this interaction before putting in the order” (Cardiologist 1) “…in our program we are told to be very comfortable with reaching out to pharmacy whenever we have a question …and it's something you can do while you're doing other things” (Resident 2) “My base is always just to call pharmacy because I find that it's a lot easier for me to just talk it through with somebody that knows what they're doing.” (Attending 2) “As a resident, I would probably discuss this with attending, the fellow, and then also maybe just talk with pharmacy about how much of a risk…” (Resident 1) “So I probably would not order this for her [patient] and talk more with Sr. Resident.” (Intern) “So I'm adding these medications, none of which I am familiar with at all...I mean, honestly, before I'm probably doing any of this, I am going to Lexicomp, double-checking… Again, there's no dose-checking, it seems here [alert window], so if I hadn't double-checked specific doses before, I would definitely go to Lexicomp and double-check” (Attending 3) “I think if it's a straightforward dosing question, or if it's a, could this be a side effect question, I will use Lexicomp. Like if it's just going to be a quick answer that I think I can get easily” (Attending 3) “I would go ahead and place the orders, but I would talk to either the coagulation or the cardiology service about how much of a risk that is…” (Resident 1) “…So talking with the neurologist to help make a good decision in that case” (Cardiologist 4) “I think from the work flow standpoint it's much more efficient for people to click on a Lexicomp or UpToDate as long as you trust that resource. For major decisions I think I probably would still refer to a pharmacist.” (Cardiologist 1) “From the work flow standpoint, it's much more efficient for people to click on a Lexicomp or UpToDate” (Cardiologist 1) “And I actually might go back to the mom and verify that he's taking both of these medications” (Attending 4) “…if it's something that I have comfort with, so say a rash to a cephalosporin or more commonly what we see is a rash to amoxicillin when I'm trying to write a cephalosporin that I feel fairly confident with and can… have usually had the conversation with the family ahead of time and found out what that rash is, is it one of these vague potential non allergies or is it Stevens-Johnsons. And so if it's a big potential non allergy that's very mild that I've already discussed with the family, I would go ahead and kind of override it. If not or if I'm not sure, again, I would probably kind of go back to the family and find out what the allergy was and how concerned I need to be” (Attending 2) “So I'd go ask the parent, how much is he actually getting?” In a follow-up to the same scenario, the neurologist added “I usually trust what the parents are saying unless there is some reason not to” (Neurology 3)

Abbreviations: EKG, electrocardiogram; SSRI, selective serotonin reuptake inhibitor.

Alert Timing and Design

Thirteen of 14 participants (93%) expressed a preference to see an interaction alert early in the process of placing a medication order, while seeing an alert at the end of the order is disruptive. “I want to have the alert as soon as I select the prescription” (Neurologist). “I don't want to go through the whole ordering and calculation and decision process … if I need to think first and maybe change my plans” (Neurologist).

In relation to alert window design, improving readability was mentioned by participants most frequently. “It [alert window] could be better formatted… And if I was going to redesign it, I would emphasize the names of the drugs. I would capitalize and bold them and put major contraindicated as a clear header line” (Cardiologist).

Unmet Informational Needs

Several clinicians expressed the need to see information about the severity of DDIs and patient-specific information. “…there was insufficient information to meet my information needs on the current alert… I'd want to know how common is that risk, is it something that typically overrides” (Attending physician). “I'd look up exactly what the contraindication is for … and get an EKG for the methadone to get a baseline before starting it [new medication]” (Attending). A cardiologist noted about a patient with enterococcus that before placing a new order for linezolid, they would “go back to look at her [patient], probably her microbiology.”

Additional Sources of Information

Themes that emerged from the data analysis included relying on additional sources of information beyond what is provided in an alert: specialists, electronic databases, and family members.

Specialists

Consulting pharmacists: “I would probably call Pharmacy before putting in that order to make sure it looks okay. And I would probably do as they say. But I am not going to put it in (the order) without calling them first” (Intern). [risk of serotonin syndrome] “which I have not seen before, I'm not as familiar with, so I would definitely consult pharmacy before prescribing this medication” (Attending).

Consulting with other providers: “…what I would normally do, because these are medications usually recommended by the neurologist or the epilepsy team, is kind of pause, not restart it, and wait until I have a discussion with them about that… and just kind of make a note to myself that I'm going to talk to them first before I reorder” (Attending physician). “…so I will talk to my senior resident and attending about an alternative antibiotic choice for that and not to put in the meropenem” (Intern). “So I guess I would discontinue [patient's] sertraline. That's the right thing to do. And I would have a discussion with her team to make sure that that's really the plan” (Neurologist).

Electronic Databases

The participants mentioned both the utility and limitations of electronic databases to obtain additional information about DDIs.

Utility of electronic databases: “…when I'm, you know, starting new medications or things like that I often will go to Lexicomp and do like an interaction check to see, make sure that whatever they're [patient] currently on doesn't interact or that I know what the potential interactions are… I use the Lexicomp, the interaction checker; I have it on Medscape on my phone” (Neurologist). “[I] use the order system as a first pass looking at what the standard options are and general knowledge of medicines that [they] don't use frequently…, because I've been a physician long enough that even though I don't prescribe them frequently, then I'll go to Lexicomp or Micromedex on a regular basis” (Cardiologist).

Limitations of electronic databases: “I think if it's a straightforward dosing question, or if it's a, could this be a side effect question, I will use Lexicomp. Like if it's just going to be a quick answer that I think I can get easily. For something more sophisticated, if it's a medication question, I'll talk to the pharmacy a lot because I find them to be extremely helpful. But I think Lexicomp can't help us with that sort of balancing risk-benefit, and so, you know, if linezolid is the only med that we can use, then, you know, how do we kind of work around that risk? Lexicomp really can't give me that information” (Attending physician). “…it would be nice to have that [link to interaction provided in the alert window] rather than to have to back out and go to Lexicomp or something else to figure out what the interaction is” (Neurologist).

Family Members

The interviewees talked about consulting with the families in specific instances related to home medications. “I'd talk with the family. Yeah, I'd talk with the family and answer the question for myself kind of why is he [patient] on the Omeprazole and how big of a deal it is” (Attending, on home medication's interaction with a new order). “…if I were in this scenario, I could talk to the family and say is this a medicine she [patient] has gotten before?” (Attending).

Discussion

Clinician Cognitive Workload

In this study, we engaged pediatricians in simulated case discussions of real-time encounters where they encountered DDI alerts demonstrating both the benefits of the existing CDS system and many opportunities for improving alert timing and information presentation. Both junior and experienced clinicians participated in the study and described the value of the DDI alerts and finding them to be beneficial in ensuring the safe care of their patients. More frequently, however, participants identify opportunities for improvement that align with the three of the five rights of CDS including optimizing alert timing, information presentation that provides clinically relevant information, and improving the content and clarity of information presentation.[7] These opportunities demonstrate that ongoing evolution of CDS is essential.

Clinicians work in a complex care environment where the volume of information they process in making medication-related decisions is at times overwhelming.[29] [30] They must process numerous factors including the patient's medical history, medication characteristics, side effects, interactions, and allergies. In this study, participants spoke strongly about the need for alerts to provide only necessary distraction at the right time and in the right form due to complexity and volume of information assimilated in clinical decision making. Overwhelmingly, clinicians in this study expressed a preference to have alerts presented early in the decision-making process, specifically at the time of medication selection. Clinicians expressed that early alert presentation helped to disrupt the ordering workflow at the appropriate time and would help to prevent redundant or extraneous steps such as reentering orders if the alert is presented at the time of order submission. These findings reinforce the recommendations as set forth by Payne et al in their work.[31] The alert provides new information that should be introduced while the clinician's attention is focused on medication selection.[5]

Furthermore, enough information needs to be included in the alert for the clinician to make an informed decision without being “clunky” or hard to read.[31] Poor alert presentation was identified as a barrier to optimal clinical efficiency, a finding which echoes research by Russ et al and Seidling et al, who found that quality of display was significantly correlated with user's acceptance of alerts.[10] [32] Thus, this study reinforces the need to enhance clinician workflows by providing smart alerts that offer an appropriately distracting, time-effective, and efficient step within the order entry process.

Our findings raise a more fundamental question about the role information technology should play in CDS. Effective CDS systems designed to aid providers and to enhance patient safety play an important role in clinical decision making, but there is still room for improvement.[33] Integration of medical technology and human factors represent a particular challenge for developers of CDS as not all the factors which go into designing effective CDS systems are well known.[33] [34] This study reinforces the need for designing CDS systems capable of providing clinicians with relevant, targeted, and smart information addressing their needs.

Informed Decision Making

Alerts serve the function of providing information to support medical decision making. Clinicians use heuristic-systematic information processing to assess risk and make clinical judgments and subconsciously apply these principles to alert handling.[35] Providers in this study used these skills to assess information presented in alerts and gaps in alert design. Often additional directive information was needed, such as severity of DDI presented, frequency of interaction occurring, alternative medications, and patient-specific information. These findings are supported by prior studies.[36] [37] [38]

Also, notably, the clinicians in this study specialized in a pediatric discipline. Infants and children differ from adults in their drug processing requiring a developmental approach to drug response.[39] Pediatricians in this study echoed findings by Rödle et al where comprehensive information was needed with additional drug safety functionality such as medication alternatives.[40] In addition, participants in this study were enrolled using purposeful sampling to ensure a broad range of expertise and clinical backgrounds in the field of pediatrics. Consistently the participants required additional information in a bidirectional flow with pharmacists, senior clinicians, and specialists in a field for higher order clinical decision making. Notably, patients and families were rarely mentioned in the clinical decision-making process. However, parents and families need to be viewed as valuable partners in medical decision making.[41]

In this study, clinicians demonstrated lower comfort levels with drug interactions of medication they do not frequently order. Somewhat unexpected was that the majority of the participants felt uncomfortable understanding interactions associated with medications they do frequently order (Cases 2 and 4). These findings may be the result of generally limited knowledge of DDIs by providers and reinforces that they will access that knowledge at the point of ordering by reviewing electronic references or engaging content specialists.[42] It also reiterates the importance of the design consideration of proximity of spatial and temporal proximity of the information within the alert and that such information should be placed in a manner that is intuitive so providers can readily access it.

Limitations

This study is a qualitative analysis of in person semistructured interviews. Participants were observed in real time during review of alerts and their alert handling in a simulated environment, which may result in biased receipt of information. The alerts presented to providers were major contraindicated alerts reflecting the EHR of this institution which likely influences their willingness to accept the alerts. The participants themselves were a small cohort of volunteers and may also have influenced the results. However, understanding the information needed to enhance the transmission of information remains the critical aspect under study. In addition, the study examined a small sample size of clinicians whose alert handling may not be complete.

Conclusion

Clinicians find value in the provided DDI CDS but identify the need to continue to evolve the CDS presentation with regard to alert timing, visual display, and alert content.

Clinical Relevance Statement

Clinical decision support provides important information to prescribers during the order entry process. Drug–drug interaction alerts provide important information to clinicians to help prevent harm from reaching patients. While clinicians find these alert valuable, the timing, presentation, and content of the alerts can be evolved to enhance the user experience.

Multiple Choice Questions

1. Clinicians found value in interruptive DDI alerts when they included the following information:

   • Interaction severity.

   • Alternative medications.

   • Provide an opportunity to focus during the order entry process.

   • All of the above.

   Correct Answer: The correct answer is option d, all of the above. This study found that interruptive DDI alerts are most valuable to clinicianswhen they interrupt the order entry process allowing clinicians to pause and focus during their workflow but also when they provide information that helps to facilitate next steps in clinical decision making. By providing the interaction severity the clinician can help to interpret the risk–benefit assessment of the DDI for their individual patient and additional supportive information about alternative medications can help to lead to their next step in care.

2. Opportunities to improve interruptive DDI alerts include all of the following except:

   • Timing of alert presentation.

   • Severity of the DDI.

   • Lengthy description of the interaction.

   • Frequency of DDI occurring.

   Correct Answer: The correct answer is option c, lengthy description of the interaction. Clinicians provided instructive feedback related to alert presentation and content. They identified that alerts were helpful if they were provided in logical step in the order entry process—medication selection. Alerts were felt to be supportive of clinical decision making and could be further enhanced by including directed information about the severity and frequency of the DDI occurring. However, a careful balance of providing concise, targeted information was felt to be important to help prevent displays of lengthy descriptions that the clinician may not fully read in the context of their busy clinical care setting.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Boston Children's Hospital Institutional Review Board.

• References

• 1 Kirkendall E, Kouril M, Dexheimer J. , et al. Automated identification of antibiotic overdoses and adverse drug events via analysis of prescribing alerts and medication administration records. J Am Med Inform Assoc 2017; 24 (02) 295-302
• 2 Saxena K, Lung BR, Becker JR. Improving patient safety by modifying provider ordering behavior using alerts (CDSS) in CPOE system. AMIA Ann Symp Proc 2011; 2011: 1207-1216
• 3 van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008; 15 (04) 439-448
• 4 Bryant AD, Fletcher GS, Payne TH. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inform 2014; 5 (03) 802-813
• 5 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631
• 6 Edrees H, Amato MG, Wong A, Seger DL, Bates DW. High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: override appropriateness and adverse drug events. J Am Med Inform Assoc 2020; 27 (06) 893-900
• 7 Sirajuddin AM, Osheroff JA, Sittig DF, Chuo J, Velasco F, Collins DA. Implementation pearls from a new guidebook on improving medication use and outcomes with clinical decision support. Effective CDS is essential for addressing healthcare performance improvement imperatives. J Healthc Inf Manag 2009; 23 (04) 38-45
• 8 Miller K, Mosby D, Capan M. , et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018; 25 (05) 585-592
• 9 Wong A, Amato MG, Seger DL. , et al. Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit. BMJ Qual Saf 2018; 27 (09) 718-724
• 10 Seidling HM, Phansalkar S, Seger DL. , et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011; 18 (04) 479-484
• 11 Seidling HM, Klein U, Schaier M. , et al. What, if all alerts were specific - estimating the potential impact on drug interaction alert burden. Int J Med Inform 2014; 83 (04) 285-291
• 12 Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc 2014; 21 (e2): e332-e340
• 13 Baysari MT, Lowenstein D, Zheng WY, Day RO. Reliability, ease of use and usefulness of I-MeDeSA for evaluating drug-drug interaction alerts in an Australian context. BMC Med Inform Decis Mak 2018; 18 (01) 83
• 14 Cho I, Lee J, Han H, Phansalkar S, Bates DW. Evaluation of a Korean version of a tool for assessing the incorporation of human factors into a medication-related decision support system: the I-MeDeSA. Appl Clin Inform 2014; 5 (02) 571-588
• 15 Marcilly R, Ammenwerth E, Vasseur F, Roehrer E, Beuscart-Zéphir MC. Usability flaws of medication-related alerting functions: a systematic qualitative review. J Biomed Inform 2015; 55: 260-271
• 16 Glassman P, Miller CE. Effect of preventive dentistry training program for caregivers in community facilities on caregiver and client behavior and client oral hygiene. N Y State Dent J 2006; 72 (02) 38-46
• 17 Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. ; with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17 (01) 36
• 18 van der Sijs H, Mulder A, van Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf 2009; 18 (10) 941-947
• 19 Ivanovska V, Rademaker CM, van Dijk L, Mantel-Teeuwisse AK. Pediatric drug formulations: a review of challenges and progress. Pediatrics 2014; 134 (02) 361-372
• 20 Glaser BG. Theoretical Sensitivity: Advances in the Methodology of Grounded Theory. Mill Valley, CA: Sociology Press; 1978
• 21 Maxwell JA. Qualitative Research Design: An Interactive Approach. Vol. 41. Thousand Oaks, CA: Sage Publications, Inc.; 1996
• 22 Yin RK. Case Study Research: Design and Methods. 4th ed. Thousand Oaks, CA: Sage; 2009
• 23 Aitken LM, Mardegan KJ. “Thinking aloud”: data collection in the natural setting. West J Nurs Res 2000; 22 (07) 841-853
• 24 Olson G, Duffy S, Mack R. New Methods in Reading Comprehension Research. Hillsdale, NJ: Erlbaum; 1984
• 25 Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. Thousand Oaks, CA: Sage; 1994
• 26 Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Thousand Oaks, CA: Sage Publications, Inc.; 1998
• 27 Glaser B, Strauss A. The Discovery of Grounded Theory: Strategies for Qualitative Research. Mill Valley, CA: Sociology Press; 1967
• 28 Crabtree BF, Miller WL. Doing Qualitative Research. 2nd ed. Thousand Oaks, CA: Sage Publications; 1999
• 29 Reason J. Understanding adverse events: human factors. Qual Health Care 1995; 4 (02) 80-89
• 30 Treiber LA, Jones JH. Devastatingly human: an analysis of registered nurses' medication error accounts. Qual Health Res 2010; 20 (10) 1327-1342
• 31 Payne TH, Hines LE, Chan RC. , et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc 2015; 22 (06) 1243-1250
• 32 Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Ann Symp Proc 2009; 2009: 548-552
• 33 Miller A, Moon B, Anders S, Walden R, Brown S, Montella D. Integrating computerized clinical decision support systems into clinical work: a meta-synthesis of qualitative research. Int J Med Inform 2015; 84 (12) 1009-1018
• 34 Lobach DF. The road to effective clinical decision support: are we there yet?. BMJ 2013; 346: f1616
• 35 Trumbo CW. Heuristic-systematic information processing and risk judgment. Risk Anal 1999; 19 (03) 391-400
• 36 Kuperman GJ, Bobb A, Payne TH. , et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14 (01) 29-40
• 37 Garg AX, Adhikari NK, McDonald H. , et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
• 38 Paterno MD, Maviglia SM, Gorman PN. , et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 (01) 40-46
• 39 Kearns GL, Abdel-Rahman SM, Alander SW, Blowey DL, Leeder JS, Kauffman RE. Developmental pharmacology--drug disposition, action, and therapy in infants and children. N Engl J Med 2003; 349 (12) 1157-1167
• 40 Rödle W, Wimmer S, Zahn J. , et al. User-centered development of an online platform for drug dosing recommendations in pediatrics. Appl Clin Inform 2019; 10 (04) 570-579
• 41 Fiks AG, Jimenez ME. The promise of shared decision-making in paediatrics. Acta Paediatr 2010; 99 (10) 1464-1466
• 42 Ko Y, Malone DC, Skrepnek GH. , et al. Prescribers' knowledge of and sources of information for potential drug-drug interactions: a postal survey of US prescribers. Drug Saf 2008; 31 (06) 525-536