Interprofessional Evaluation of a Medication Clinical Decision Support System Prior to Implementation

Abstract

Background Computerized physician order entry (CPOE) and clinical decision support systems (CDSS) are widespread due to increasing digitalization of hospitals. They can be associated with reduced medication errors and improved patient safety, but also with well-known risks (e.g., overalerting, nonadoption).

Objectives Therefore, we aimed to evaluate a commonly used CDSS containing Medication-Safety-Validators (e.g., drug–drug interactions), which can be locally activated or deactivated, to identify limitations and thereby potentially optimize the use of the CDSS in clinical routine.

Methods Within the implementation process of Meona (commercial CPOE/CDSS) at a German University hospital, we conducted an interprofessional evaluation of the CDSS and its included Medication-Safety-Validators following a defined algorithm: (1) general evaluation, (2) systematic technical and content-related validation, (3) decision of activation or deactivation, and possibly (4) choosing the activation mode (interruptive or passive). We completed the in-depth evaluation for exemplarily chosen Medication-Safety-Validators. Moreover, we performed a survey among 12 German University hospitals using Meona to compare their configurations.

Results Based on the evaluation, we deactivated 3 of 10 Medication-Safety-Validators due to technical or content-related limitations. For the seven activated Medication-Safety-Validators, we chose the interruptive option [“PUSH-(&PULL)-modus”] four times (4/7), and a new, on-demand option [“only-PULL-modus”] three times (3/7). The site-specific configuration (activation or deactivation) differed across all participating hospitals in the survey and led to varying medication safety alerts for identical patient cases.

Conclusion An interprofessional evaluation of CPOE and CDSS prior to implementation in clinical routine is crucial to detect limitations. This can contribute to a sustainable utilization and thereby possibly increase medication safety.

 

Scroll to top

Keywords

clinical decision support system - clinical pharmacists - evaluation - medication safety - alert fatigue

Background and Significance

Since “To Err is human—Building a Safer Health System” was published, medication errors and patient safety are on public agenda.[1] As part of the digitalization in hospitals to improve medication safety, the implementation and usage of computerized physician order entry (CPOE) and clinical decision support systems (CDSS) is internationally widespread.[2] [3] [4] [5] Although the use of electronic medical records (EMR) has increased in German hospitals within the last years, they are not utilized nationwide.[6] [7]

CPOE systems allow physicians to prescribe medication orders in a direct, electronical way.[8] CDSS are often integrated into CPOE systems, but also stand-alone CDSS are available.[9] [10] CDSS can be categorized regarding how warnings and recommendations are presented: interruptive or passive/on-demand.[10] The operating mode of a CDSS can be rule-based or without using a rule (e.g., artificial intelligence, neural networks, machine learning).[2] [9] [10] Hospitals use commercial CDSS as well as homegrown systems.[9] CDSS can address various different topics in clinical routine (e.g., medication safety, diagnostic support, guideline adherence).[2] [11]

CPOE and CDSS can reduce medication errors, and thereby optimize medication safety.[12] [13] [14] [15] [16] [17] [18] However, the implementation of a CDSS is also related with risks like alert fatigue and nonadoption of the system among the users.[19] [20] [21] [22] [23] To minimize overalerting and to foster acceptance among health care professionals, the implementation should be well-prepared by analyzing the CPOE and CDSS in detail prior to their roll-out.[10] [24] [25] Many studies, which dealt with customizing a medication CDSS so far, used quantitative outcomes from the postimplementation phase (e.g., overridden rates).[26] [27] [28] [29] To date, little has been published on the methodological approach for a preimplementation evaluation of a medication CDSS.[30]

 

Scroll to top

Objectives

We set out to develop a general algorithm for a preimplementation-evaluation process of a medication CDSS. We present the results for one commercial, German CPOE with an integrated medication CDSS, for which no comprehensive data are yet available. CDSS can often be customized according to local circumstances. This may lead to different safety alerts across various sites. Therefore, the results of our evaluation were compared with the selected configurations of the CDSS in other German University hospitals using the same system.

 

Scroll to top

Methods

Software

Meona (Mesalvo Freiburg GmbH, Germany) is a commercial EMR and is registered as a medical device. It is a CPOE system with an integrated medication CDSS (rule-based with interruptive as well as passive alerts/recommendations).[31] The single elements of the medication CDSS are called “Medication-Safety-Validators.” There are 19 different Medication-Safety-Validators available, each addressing a different topic (e.g., “drug–drug interactions”). A list of all Medication-Safety-Validators is provided in [Supplementary Appendix A1] (available in the online version). For the present evaluation of the CDSS, a daily updated test system of Meona was used to simulate clinical scenarios.[32]

 

Scroll to top

Setting

As a large academic medical center, the Erlangen University Hospital consists of 51 medical departments and 57 interdisciplinary centers.[33] Since June 2020, Meona was rolled out step-by-step to all standard care units. The wards utilize Meona for the documentation of clinical processes and values (e.g., medication prescription, care documentations). Besides different clinical departments and the Medical Center for Information and Communication Technology (MIK), the Pharmacy Department was a key pillar of the EMR implementation project team in Erlangen.

 

Scroll to top

Interprofessional Evaluation of an Integrated Medication CDSS

For the evaluation process, we followed the algorithm presented in [Fig. 1]. This algorithm was developed and determined in an interprofessional team by considering available literature[10] [25] [34] [35] [36] [37] and contained four consecutive steps:

• Step 1—General CDSS evaluation: this consists of analyzing the general functionalities and structure of the CDSS. If the first section of the process is not successfully passed, further improvements of the CDSS (not yet focusing on single elements of the CDSS, see step 2) will be required and initiated with the development department of the CDSS supplier before implementation will be continued.

• Step 2—Technical and content-related validation: the single elements of the CDSS (e.g., Medication-Safety-Validators) are checked for their technical operating mode and technical limitations (a.). Furthermore, a systematic content-related validation of the Medication-Safety-Validators is performed (b.). This approach strongly differs throughout the Medication-Safety-Validators due to varying operating modes and context parameters of every Medication-Safety-Validator (see [Supplementary Appendix A2] [available in the online version]).

• Step 3—Decision of activation: this comprises presenting and discussing the results interprofessionally to decide upon the activation or deactivation of the Medication-Safety-Validators. The decisions were made by majority vote, for details of the process and organization, see [Supplementary Appendix A3] (available in the online version). Further enhancements have to be initiated for all deactivated Medication-Safety-Validators.

• Step 4—Decision of activation mode: the decision of activation is finally followed by another discussion determining the activation mode (interruptive or passive/on-demand).

Fig. 1 Algorithm for the interprofessional evaluation of an integrated CDSS. CDSS, clinical decision support system.

In Erlangen an interprofessional working group of the Medicines Management Board was established to evaluate the integrated CDSS and its included Medication-Safety-Validators as displayed in the developed algorithm ([Fig. 1]). The expert panel included 15 physicians, 6 clinical pharmacists, 2 nurses, and 2 IT-experts. The functionalities, possibilities, limitations, advantages, and disadvantages of the CDSS (i.e., each Medication-Safety-Validator) were discussed in regular meetings. For performing the second, third, and fourth step of our algorithm, we exemplarily selected Medication-Safety-Validators in a consensus-based way after interprofessional discussion. Thereby, we included different aspects in the selection process: (1) clinical relevance, (2) possible impact in clinical practice, and (3) local factors and processes. If in any step throughout the algorithm, the result was that further improvements had to be implemented before the CDSS or the individual validators could be utilized, the working group formulated optimizations. These were forwarded to the Meona development department for realization.

 

Scroll to top

Survey among German University Hospitals Using Meona

Throughout the implementation process of the EMR in our institution, we performed an online survey among all 12 German University sites utilizing Meona (including Erlangen) from a total of 38 University hospitals in Germany. The questionnaire was generated interprofessionally by clinical pharmacists and IT-experts. The survey consisted out of self-compiled questions and covered three main topics: (1) implementation and usage status of Meona, (2) aspects of the collaboration of the Pharmacy Department within the implementation, and (3) site-specific configuration of the CDSS (Medication-Safety-Validators).

The questionnaire was sent via email and contained a personal link for each site for single use (survey period: 6 weeks, reminder after 4 weeks). The online survey was performed with SoSci Survey[38] and the descriptive data analyses were performed in Microsoft Excel. All results are presented in a pseudonymized form—except the dataset of Erlangen.

 

Scroll to top

 

Scroll to top

Results

Interprofessional Evaluation of an Integrated Medication CDSS

Applying the developed algorithm ([Fig. 1]) to Meona resulted in the following findings:

• Step 1—General CDSS evaluation: we assessed that operating modes for each Medication-Safety-Validator were limited to two configurations: activation or deactivation. The activation of a Medication-Safety-Validator always resulted in interrupting popup alerts (“PUSH-modus”), which can be invoked later by every health care professional via a check button (“PULL-modus”). Due to the high potential of overalerting, further improvements before using the CDSS in clinical routine were inquired. All advancements are shown in [Fig. 2] as a comparison before and after evaluation. As a major improvement, the new configuration option “only-PULL,” requested by our working group, was available since April 2021 for the Medication-Safety-Validators which are displayed via the so-called “check button” (for details see [Supplementary Appendix A1] [available in the online version]). Another substantial improvement was the disclosure of the activated Medication-Safety-Validators in the display of the check button: As a result, every health care professional is now informed about the examined medication safety aspects to avoid a false sense of security by relying on the CDSS performing an all-embracing medication review. Additionally, the possibility to filter the severity level of the shown alerts was added in the display of the check button to further reduce overalerting. An example of a PUSH alert and the composition of the revised display of the check button are included in [Supplementary Fig. S1] (available in the online version). The initiated improvements were made available for all Meona customers, including other sites.

• Step 2—Technical and content-related validation: for the second to fourth step, the interprofessional working group chose 10 Medication-Safety-Validators ([Table 1]) in a consensus-based way. For example, we chose “drug–drug interactions,” “allergy,” “maximum daily dose under consideration of the kidney function”, and “duplicate prescription” due to their high clinical relevance and expected positive impact in daily clinical routine. The interprofessional working group decided not upon the Medication-Safety-Validator “indications and contraindications,” because diagnoses in Erlangen are currently not recorded within the EMR by using the ICD coding (International Statistical Classification of Diseases and Related Health Problems) during routine inpatient care. [Table 1] presents the results of the technical and content-related validation considering the operating mode of the Medication-Safety-Validators, the references used to create the alert, the included context parameters, and our identified limitations. As an illustration, the technical limitation of the Medication-Safety-Validator “maximum daily dose under consideration of the kidney function” was that there was no possibility to sum up the dose for more than one prescription line. In conclusion, no alert would be displayed if ibuprofen 600 mg 1–1–1–1 and ibuprofen 400 mg 1–1–1–0 were prescribed although the maximum daily dose was exceeded.

• Step 3—Decision of activation: following the results of step 2 within the interprofessional working group, we decided to activate seven and to deactivate three Medication-Safety-Validators ([Fig. 3]). There were different reasons for the decision of deactivation: (1) technical limitations (e.g., “incompatibilities,” “maximum daily dose under consideration of the kidney function”), (2) content-related limitations (e.g., “maximum daily dose under consideration of the kidney function”), and (3) prevention of overalerting and increased convenience (e.g., “feeding tube information”).

• Step 4—Decision of activation mode: the seven activated Medication-Safety-Validators were further divided into four using the “PUSH-(&PULL)-modus” and three using the “only-PULL-modus” ([Fig. 3]). There are some Medication-Safety-Validators for which the “PUSH-(&PULL)-modus” is the only useful configuration, as the alert is urgent with high clinical relevance (e.g., “allergy”). For instance, the alert that ampicillin might be contraindicated for a patient with a documented allergy to amoxicillin must be assessed immediately to prevent a potential serious adverse event (e.g., anaphylaxis). Even though we decided to use the “PUSH-(&PULL)-modus” for a Medication-Safety-Validator, there are still limitations to be considered. For example, a duplicate prescription alert is only created if the medication is prescribed for the same daytime. A prescription of ramipril in the morning and enalapril in the evening would not create an alert. The working group decided to use the “only-PULL-modus” for Medication-Safety-Validators with a high risk of creating overalerting (e.g., “drug–drug interactions”). Furthermore, the “only-PULL-modus” is suitable for Medication-Safety-Validators for which realization of the recommendation is not always urgent, but essential for comprehensive medication reviews (e.g., “information about renal impairment”). For example, dose adjustments in renal impairment for several medications (e.g., anti-infective drugs) are only relevant after administering a full loading dose.

Fig. 2 Configuration options of the CDSS before and after the evaluation by initiating some improvements. After evaluation means after performing the first step of the algorithm presented in [Fig. 1]. “PUSH” means that a popup alert is automatically displayed during the prescription. “PULL” means that alerts can be invoked via the “check button” at any time (passive/on-demand alerts). The “check button” can be found in the medication chart. The configuration option “only-PULL” can only be selected for Medication-Safety-Validators, which are presented in the “check button”. For detailed information, see [Supplementary Appendix A1] (available in the online version).

Table 1 Operating modes, references, context parameters, and limitations of the 10 exemplarily evaluated Medication-Safety-Validators

	Name Medication-Safety-Validator	Operating mode/display of an alert	References	Limitations/reasons for actual configuration
			Context parameters
1	“Divisibility information”	▪ By prescription of a nondivisible medication	Meona-database based on SmPCs or manufacturing information	➢ None
2	“Duplicate prescription”	▪ By prescription of two medications with the same ATC code (5 characters) at the same daytime	ATC code	➢ Medication on demand is not included ➢ Display of the alert only by prescription for the same daytime (e. g., both 8 am) ➢ No consideration of relevant duplicate prescription with different ATC code (5 characters, e.g., apixaban and enoxaparin)
3	“Allergy”	▪ By prescription of the same or a similar drug with a statement to allergy or possible cross-reactivity	Meona-database with substance-specific allergy codes	➢ Risk of overalerting by registering of clinical nonvalid allergy
4	“Frequency of administration”	▪ By underrunning the recommended dosage interval for the actual prescription	Meona-database based on SmPCs or manufacturing information	➢ Solely checking of the current prescription (e.g., Methotrexate 10mg po OD versus every 7 days), no check for already prescribed or administrated dosages
5	“Drug–drug interactions”	▪ By onset of a drug–drug interaction (binary combination) ▪ PUSH alerts only for severe interactions	Meona-database considering inter alia Stockleys Interaction, the ABDA-Database, crediblemeds, and SmPCs	➢ Risk of overalerting by: ➢ Strong weighting of QT–time interactions, because the interaction is always severe (not depending on the classification of the drug as “known risk,” “possible risk,” or “risk under certain condition”) ➢ Display of clinically nonrelevant interactions as severe alerts (e.g., electrolyte solution and candesartan)
6	“Information about renal impairment”	▪ By underrunning as per drug defined limit of the eGFR (“consider the hints about renal insufficiency,” no concrete dosage suggestion)	Meona database considering inter alia BNF, SmPCs, Renal Drug Handbook, and dosing.com	➢ Display of the alert “consider the hints about renal insufficiency” is only based on the actual GFR, not on the actual prescribed dosage ➢ No hint about administering unreduced loading doses for anti-infective drugs at the beginning of the therapy
			Creatinine level (eGFR according to MDRD)
7	“Priscus-/Forta-list of potentially inadequate medication for the elderly”[a]	▪ By PIM prescription for patients >65 years, stating alternative drugs or measures if the prescription is necessary (e.g., monitoring)	Priscus Report 1.0 or. Forta list	➢ Content-related revision is necessary (Priscus 2.0)
			Patient age
8	“Drug incompatibilities”	▪ By prescription of two incompatible drugs simultaneously	Meona-database considering inter alia ASHP: Handbook on injectable Drugs, and SmPCs	➢ Display of an alert for short infusions only if prescribed for simultaneous daytime (with accuracy of 1 minute) leads to a lack of relation for clinical practice ➢ No possibility of checking the incompatibilities for nursing staff easily
9	“Maximum daily dose under consideration of the kidney function”	▪ By exceeding the maximum daily dose, also depending on the actual eGFR (individual dose calculation according to Dettli under consideration of the Q0-value)	Inter alia SmPCs	➢ No possibility to sum up the maximum daily dose over more than one prescription line ➢ Individual dose calculation is not practicable in clinical routine (e.g., sitagliptin 46.8 mg)
			Actual prescribed dose, creatinine level (eGFR according to MDRD), Q0-value
10	“Feeding tube information”	▪ By prescription of a drug, which cannot be administered via a feeding tube	Meona-database based on SmPCs or manufacturing information	➢ Checking for feeding tube information can be done directly at the tube prescription in Meona for all prescribed medications simultaneously for nursing staff and physicians
			Documentation of a feeding tube in Meona

Abbreviations: ABDA-database, German database containing all approved medications and further information, e.g., drug–drug interaction check; ATC code, Anatomical Therapeutic Chemical; BNF, British National Formulary; DOAC, direct oral anticoagulant; eGFR, estimated glomerular filtration rate; FORTA, Fit for the Aged; MDRD, Modification of Diet in Renal Disease; PIM, potential inadequate medication; OD, once daily; Q₀-value, extrarenally metabolized proportion; SmPC, Summary of Product Characteristics.

^a Priscus and FORTA lists are lists for potentially inadequate medication for older patients that are frequently used in Germany/Europe. The Priscus list is a negative list and is comparable to the BEERS criteria[60] in the United States. In contrast, the FORTA list comprised drugs according to the indication, from “highly recommended” (rated with A) to “to be avoided” (rated with D) and is rather comparable to the START/STOPP criteria.[61]

Fig. 3 Results of the interprofessional evaluation of the integrated CDSS (Meona) at Erlangen University Hospital. Overview of the results of the evaluation from the second to fourth step presented in [Fig. 1]. The exemplary Medication-Safety-Validators were selected in a consensus-based way. PUSH means that a popup alert is displayed during the prescription. PULL means that alerts can be invoked via the check button at any time (passive/on-demand alerts). CDSS, clinical decision support system.

 

Scroll to top

Survey among German University Hospitals Using Meona

In total, 12 University hospitals in Germany using Meona responded to the survey (response rate: 100%). However, the dataset of one site had to be excluded in this analysis because Meona was not yet used as an EMR.

The Erlangen University Hospital provided 1,450 beds in comparison to an average number of 1,495 beds [924–2,600] among the other sites. All 11 sites used Meona in their standard care units as EMR and every site confirmed the collaboration of the Pharmacy Department within rollout and maintenance process in the survey (for details see [Table 2]). On average, the sites selected 8 out of 10 (80%) tasks for pharmacists working in the project team like testing new Meona versions, performing training sessions, as well as assuming medication configurations (e.g., creating medication order templates). All possible tasks are presented in [Supplementary Appendix A4] (available in the online version).

Table 2 Structural characteristics as a comparison between Erlangen and all other University hospitals using Meona

“Meona in intensive care units” means for example that it is implemented in at least one ward within the respective sites.

The configuration (e.g., activated vs. deactivated) of the Medication-Safety-Validators was very heterogeneous across the participating sites ([Table 3]). For example, site 1 and 7 activated 94.7% (18/19) of the possible Medication-Safety-Validators, whereas site 6 activated only 21.1% (4/19) and site 11 (Erlangen) activated 36.8% (7/19) of all Medication-Safety-Validators. Erlangen configured the on-demand option “only-PULL” for three Medication-Safety-Validators exclusively (“drug–drug interactions,” “information about renal impairment,” and “Priscus-/Forta-list of potentially inadequate medication for the elderly”). Rarely, all sites used the same configuration (e.g., deactivation: “interactions with alcohol,” activation: “divisibility information”). To sum up, 67.5% (141/209) of all configuration options used the “PUSH-(&PULL)”-modus and 31.1% (65/209) used “OFF-modus.” The “only-PULL-modus” was utilized for the minority of 1.4% (3/209) in our survey.

Table 3 Overview of the configuration of all Medication-Safety-Validators among the different sites

A) Medication-Safety-Validators with the configuration options “PUSH-(&PULL),” “only-PULL,” and “OFF.”

B) Medication-Safety-Validators with the configuration options “ACTIVE” or “OFF.”

The different University hospitals are sorted according to their ascending full-time equivalents for the collaboration of the Pharmacy Department within the rollout and maintenance process of Meona. Number 11 represents Erlangen University Hospital.

^# Medication-Safety-Validators which were exemplary validated in Erlangen (see section: Results - Interprofessional Evaluation of an Integrated Medication Clinical Decision Support System).

 

Scroll to top

 

Scroll to top

Discussion

In our investigation, we outlined the process of thoroughly evaluating and validating a CDSS prior to its implementation into clinical practice at a large German University hospital. We analyzed the commercially available and among German University hospitals commonly used Meona-CDSS with a special focus on the integrated Medication-Safety-Validators. Based on a predefined algorithm ([Fig. 1]), an interprofessional team assessed the general functionalities of the CDSS as well as the technical and content-related limitations for each checked Medication-Safety-Validator and identified barriers for a sustainable implementation.

Data on the usefulness of Meona as a medication CDSS in clinical routine are scarce.[39] [40] [41] For instance, Amkreutz et al[40] only performed an evaluation of the drug–drug interaction check but none of the investigations performed a detailed evaluation of the entire medication CDSS integrated in Meona.

Several investigations of other CDSS (e.g., AiDKlinik[42]) in Germany focused on quantitative outcomes solely or assessed only partial aspects of medication safety (e.g., overdose, drug–drug interactions).[43] [44] Various German studies evaluated different drug–drug interaction checks.[40] [45] To the best of our knowledge, we performed the first comprehensive evaluation of the Meona-medication-CDSS and of an entire medication CDSS (integrated in a CPOE) in Germany.

In contrast to other publications and recommendations focusing on quantitative outcomes in clinical routine,[25] [46] we described a qualitative evaluation of a medication CDSS. McCoy et al established and evaluated a framework to rate the content and responder appropriateness of displayed alerts.[46] Few studies focus on the qualitative evaluation of timing and presentation of the medication alerts.[37] [47]

We identified considerable limitations during the interprofessional evaluation and validation of all analyzed Medication-Safety-Validators in Meona ([Table 1]). Our results emphasize that uncritical activation of all validators is not without risks. Thus, an evaluation prior to the implementation is indispensable.

Some of our detected limitations have been described in other investigations as well (e.g., risk of alert fatigue).[19] [20] [48] Considering overalerting as a risk for nonacceptance, several concepts have been developed and published to prevent alert fatigue and optimize overridden rates of reported alerts (e.g., tailoring the displayed alerts for selected wards or specific end-users).[43] [49] [50] [51] Since customizing specific alerts in Meona (e.g., switch off a defined drug–drug interaction) is not possible for an individual site, we developed the configuration option “only-PULL” to prevent alert fatigue. Further, we added the possibility to filter severe alerts in the display of the check button (see [Fig. 2]), as this has proven to be an additional effective strategy to reduce overalerting.[28] [52]

In our approach, technical limitations (e.g., no possibility to sum up the maximum daily dose over more than one prescription line) often resulted in the deactivation of Medication-Safety-Validators, whereas content-related limitations and the risk of overalerting led to activation of the Medication-Safety-Validators in the “only-PULL-modus.” Other studies showed that overridden rates are higher among alerts affecting drug–drug interactions, renal dose adjustments, and geriatric recommendations as opposed to alerts affecting duplicate prescription, allergy alerts, and overdose alerts.[22] [48] These findings are in line with our selected configuration after performing the evaluation (see [Fig. 3]).

Nevertheless, interruptive alerts are generally considered more effective in terms of clinical outcomes (e.g., higher acceptance rates).[10] [49] Therefore, the risk of overalerting by interruptive alerts on the one hand and potentially less effective clinical outcomes by passive alerts on the other hand must be weighed out.[51] [53] Irrespective of the final decision of activation or deactivation of a Medication-Safety-Validator, every health care professional needs to be regularly informed and educated about the selected configuration and developed improvements.[34] In addition, it is even more important that health care professionals understand the technical and content-related limitations for the purpose of preventing errors, misunderstandings, and a false sense of security.[34] Therefore, we implemented a disclosure of all active Medication-Safety-Validators in the display of the check button as a key improvement ([Fig. 2]).

The “Office of the National Coordinator for Health Information Technology” recommends to perform the evaluation of CPOE and CDSS within an interprofessional team.[25] Other investigations emphasize the involvement of key players as hybrid experts, who understand the clinical workflow as well as the technical backgrounds (e.g., clinical pharmacists).[54] [55] If technical requirements are considered from the outset, this may lead to more effective and rapid transformation and implementation of improvements into clinical practice. We integrated both recommendations as well as the integration of IT support for a more effective and rapid realization in our evaluation approach. Conducting a detailed and interprofessional CDSS evaluation is time-consuming and challenging for health care systems. Nevertheless, our study showed that technical and content-related validation is crucial to identify limitations of medication CDSS. Additionally, a postimplementation evaluation should be performed and repeated evaluations should be performed as needed, e.g., for CDSS updates.

As a constraint of our investigation, the detected limitations cannot be transferred to other CDSS, but can raise awareness for possible limitations and important factors during the evaluation of CDSS for other hospitals.[56] To the best of our knowledge, no studies have already addressed the generalizability and translation of frameworks to evaluate medication CDSS in clinical practice. In our opinion, our algorithm ([Fig. 1]) can be adopted for a detailed evaluation of other CDSS (eventually omitting the fourth evaluation step, if not applicable). Our approach is especially suitable for commercial systems, since part of the required evaluation in the algorithm is usually performed during the development of homegrown CDSS.[10]

Our technical and content-related evaluation approach has further limitations: the decisions of activation or deactivation of the Medication-Safety-Validators were broad-consensus-based, but did not undergo a Delphi process due to limited time and personnel resources. In addition, we performed the in-depth validation presented in [Fig. 1] for 10 out of 19 Medication-Safety-Validators and focused on the preimplementation process in this investigation. The pending Medication-Safety-Validators should be reviewed in the future. Even though we performed an interprofessional evaluation and strongly included health care professionals (e.g., nursing staff, physicians) in this process, the acceptance of our configuration of the Medication-Safety-Validators among the end-users in clinical routine has not been evaluated yet. As multiple studies and recommendations emphasized that user satisfaction is one of the main barriers for successful implementation of a CPOE and CDSS, further investigations and surveys should be performed to determine their adoption and acceptance.[2] [23] [25] [34]

Although we performed a content-related validation for different Medication-Safety-Validators by testing defined scenarios (see [Table 1]; [Supplementary Appendix A2] [available in the online version]) to decide on the activation or deactivation of the Medication-Safety-Validators in Erlangen, the evaluation has not yet been performed with any clinical routine data. Thus, further prospective studies should be conducted to evaluate the usability and relevance of the displayed alerts in Meona in clinical routine and in the context of an established clinical pharmacist service.

The results of our survey showed that the same CPOE and CDSS is diversely configured and utilized across different University hospitals in Germany ([Table 3]). The phenomenon of heterogeneous configuration and implementation of a CPOE and CDSS among various sites has been already described in the literature.[57] [58] [59] One reason for the widely varying configurations might be limited personal resources. Limited time available for the Pharmacy Department to participate in the project team may require prioritizing the most challenging and important aspects of CPOE and CDSS implementation (e.g., creating standard order templates) thereby lacking a detailed assessment of the medication CDSS as provided in our present analysis.[25] Other reasons for heterogeneous configurations should be investigated in the future.

On the one hand, the diversity of configurations among the Medication-Safety-Validators can be perceived as an advantage of Meona, as customizing the CPOE and especially the CDSS for each site (i.e., activation or deactivation of Medication-Safety-Validators) is possible according to local workflows, circumstances, and preferences. On the other hand, this may lead to different medication safety outcomes and varying acceptance of the system in clinical routine. To evaluate the performance of CPOE and CDSS, the Leapfrog methodology has been developed in the United States.[36] As a part of that assessment, every CPOE and CDSS will be rated for their performance in completing patient test cases. Adopting this method for future investigations might be a chance for hospitals in Germany to compare their CPOE and CDSS performance and might not only result in optimization, but particularly in standardization.[58]

 

Scroll to top

Conclusion

Several lessons can be learned from our preimplementation approach of evaluating a medication CDSS: the analyzed German, commercial medication CPOE and CDSS was heterogeneously implemented among different sites and revealed remaining technical as well as content-related limitations. Communicating capabilities and limitations to the end-users is a major implementation challenge to achieve the best possible performance with the CPOE and CDSS. The activation of the CDSS (i.e., Medication-Safety-Validators) should be critically and interprofessionally reviewed to outweigh possible benefits and risks for medication safety. Our customized CDSS may potentially achieve improvements in clinical practice (e.g., user acceptance, medication safety), but this needs to be proven in further investigations. However, the interprofessional evaluation led to substantial improvements of the CDSS (e.g., possibility to filter severe alerts) and the developed algorithm can serve as a guidance to evaluate and validate CDSS at other sites.

 

Scroll to top

Clinical Relevance Statement

Our research demonstrated that interprofessional evaluation of CPOE/CDSS (especially commercial systems) prior to implementation is crucial to detect remaining limitations and optimize utilization. The presented algorithm for evaluating a medication CDSS is a reliable validation approach and can be used by other sites to evaluate their system.

Our survey revealed that the same CPOE/CDSS is locally often configured heterogeneously resulting in varying medication safety alerts.

 

Scroll to top

Multiple-Choice Questions

1. What are possible reasons for the deactivation of a Medication-Safety-Validator?

   • Overalerting and technical limitations

   • Personal opinion of physicians

   • Personal opinion of clinical pharmacists

   • Alerts with high relevance in clinical practice

   Correct Answer: The correct answer is option a. Overalerting and technical limitations are risks for successful implementation and user acceptance.

2. How many Medication-Safety-Validators were deactivated in Erlangen due to limitations?

   • 1

   • 3

   • 7

   • 10

   Correct Answer: The correct answer is the option b. We deactivated the Medication-Safety-Validator “incompatibilities,” “maximum daily dose under consideration of the kidney function,” and “feeding tube information.”

3. How many sites in the survey use the configuration option “only-PULL” for a Medication-Safety-Validator?

   • 10

   • 7

   • 4

   • 1

   Correct Answer: The correct answer is option d. Erlangen developed and configured the option “only-PULL” exclusively to reduce overalerting.

 

Scroll to top

 

Scroll to top

Conflict of Interest

M.F.F.: consulting or advisory roles (Boehringer Ingelheim); research funding (Boehringer Ingelheim, Heidelberg Pharma Research GmbH); other relationship (earmarked financial contribution for the first award of the MSD Germany Health Award 2021).

F.D.: honoraria (lecture fees from E. Lilly); consulting or advisory roles (Boehringer Ingelheim, Lilly Deutschland, Pfizer Pharma GmbH, SANDOZ AG); other relationship (earmarked financial contribution for the first award of the MSD Germany Health Award 2021).

All other authors declare no competing interests. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Acknowledgments

The authors gratefully acknowledge all members of the working group “Medication-Safety-Validators” of the Medicines Management Board of the Erlangen University Hospital. The authors acknowledge Lisa Cuba and Katja Gessner for critically reviewing the manuscript.

The present work was performed in (partial) fulfillment of the requirements for obtaining the degree “Dr. rer. biol. hum.” (“J.B.”).

Note

This study was conducted in collaboration with members of the working group “Medication-Safety-Validators” of the Medicines Management Board of the Erlangen University Hospital.

Protection of Human and Animal Subjects

In this project no human and/or animal subjects were included.

Authors' Contributions

J.B., F.D., and M.F.F. designed the research.

All authors are responsible for all aspects of the presented work.

J.B. analyzed all data; M.B. and T.K. verified the data analyses.

J.B. and F.D. wrote the first version of the manuscript; M.F.F, M.B., T.K., and, C.S. revised the manuscript for important content.

All authors finally approved the manuscript.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

• References

• 1 Institute of Medicine (US) Committee on Quality of Health Care in America. To Err is Human: Building a Safer Health System. Washington, DC: National Academies Press (US); 2000
• 2 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3: 17
• 3 Pedersen CA, Schneider PJ, Scheckelhoff DJ. ASHP national survey of pharmacy practice in hospital settings: prescribing and transcribing-2016. Am J Health Syst Pharm 2017; 74 (17) 1336-1352
• 4 Agency for Healthcare Research and Quality. Clinical Decision Support Systems. Updated 07.09.2019. Accessed August 11, 2023 at: https://psnet.ahrq.gov/primer/clinical-decision-support-systems
• 5 Office of the National Coordinator for Health Information Technology. National Trends in Hospital and Physician Adoption of Electronic Health Records. 06.04.2022. Accessed August 11, 2023 at: https://www.healthit.gov/data/quickstats/national-trends-hospital-and-physician-adoption-electronic-health-records
• 6 Blum DK, Löffert DS, Offermanns DM, Steffen DP. Krankenhaus Barometer 2016. 19.12.2016. Accessed August 11, 2023 at: https://www.dki.de/sites/default/files/2019-01/2016_12_19_kh_barometer_final.pdf
• 7 Blum DK, Löffert DS, Offermanns DM, Steffen DP. Krankenhaus Barometer. 2018. 12.2018. Accessed May 26, 2024 at: https://www.dki.de/fileadmin/forschungsberichte/2018_11_kh_barometer_final.pdf
• 8 Agency for Healthcare Research and Quality. Computerized provider order entry. Updated 07.09.2019. Accessed August 11, 2023 at: https://psnet.ahrq.gov/primer/computerized-provider-order-entry
• 9 Hak F, Guimarães T, Santos M. Towards effective clinical decision support systems: A systematic review. PLoS One 2022; 17 (08) e0272846
• 10 Wasylewicz ATM, Scheepers-Hoeks A. Clinical Decision Support Systems. In: Kubben P, Dumontier M, Dekker A. eds. Fundamentals of Clinical Data Science. Cham: Springer; 2019: 153-169
• 11 Sim I, Gorman P, Greenes RA. et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc 2001; 8 (06) 527-534
• 12 Bates DW, Teich JM, Lee J. et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6 (04) 313-321
• 13 Vélez-Díaz-Pallarés M, Pérez-Menéndez-Conde C, Bermejo-Vicedo T. Systematic review of computerized prescriber order entry and clinical decision support. Am J Health Syst Pharm 2018; 75 (23) 1909-1921
• 14 Garg AX, Adhikari NK, McDonald H. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
• 15 Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008; 15 (05) 585-600
• 16 Wolfstadt JI, Gurwitz JH, Field TS. et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008; 23 (04) 451-458
• 17 Nuckols TK, Smith-Spangler C, Morton SC. et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev 2014; 3: 56
• 18 Jia P, Zhang L, Chen J, Zhao P, Zhang M. The effects of clinical decision support systems on medication safety: an overview. PLoS One 2016; 11 (12) e0167683
• 19 Zaal RJ, Jansen MM, Duisenberg-van Essenberg M, Tijssen CC, Roukema JA, van den Bemt PM. Identification of drug-related problems by a clinical pharmacist in addition to computerized alerts. Int J Clin Pharm 2013; 35 (05) 753-762
• 20 Nanji KC, Seger DL, Slight SP. et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25 (05) 476-481
• 21 Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014; 23 (09) 773-780
• 22 van der Sijs H, Mulder A, van Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf 2009; 18 (10) 941-947
• 23 Abell B, Naicker S, Rodwell D. et al. Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review. Implement Sci 2023; 18 (01) 32
• 24 Ammenwerth E, Aly AF, Bürkle T. et al. Memorandum on the use of information technology to improve medication safety. Methods Inf Med 2014; 53 (05) 336-343
• 25 The Office of the National Coordinator for Health Information Technology. SAFER Self-Assessment. Computerized Provider Order Entry with Decision Support. Updated 11.2016. Accessed August 12, 2023 at: https://www.healthit.gov/sites/default/files/safer/guides/safer_cpoe.pdf
• 26 Tse G, Algaze C, Pageler N, Wood M, Chadwick W. Using clinical decision support systems to decrease intravenous acetaminophen use: implementation and lessons learned. Appl Clin Inform 2024; 15 (01) 64-74
• 27 E Dawson T, Beus J, W Orenstein E, Umontuen U, McNeill D, Kandaswamy S. Reducing therapeutic duplication in inpatient medication orders. Appl Clin Inform 2023; 14 (03) 538-543
• 28 Knight AM, Maygers J, Foltz KA, John IS, Yeh HC, Brotman DJ. The effect of eliminating intermediate severity drug-drug interaction alerts on overall medication alert burden and acceptance rate. Appl Clin Inform 2019; 10 (05) 927-934
• 29 Rehr CA, Wong A, Seger DL, Bates DW. Determining inappropriate medication alerts from “inaccurate warning” overrides in the intensive care unit. Appl Clin Inform 2018; 9 (02) 268-274
• 30 Meslin SMM, Zheng WY, Day RO, Tay EMY, Baysari MT. Evaluation of clinical relevance of drug-drug interaction alerts prior to implementation. Appl Clin Inform 2018; 9 (04) 849-855
• 31 Mesalvo Freiburg GmbH Accessed August 11, 2023 at: https://www.meona.de/
• 32 Wright A, Aaron S, Sittig DF. Testing electronic health records in the “production” environment: an essential step in the journey to a safe and effective health care system. J Am Med Inform Assoc 2017; 24 (01) 188-192
• 33 University Medical Center of Erlangen. Number and facts. Updated 02.01.2023. Accessed August 12, 2023 at: https://www.uk-erlangen.de/presse/zahlen-und-fakten/
• 34 Van de Velde S, Kunnamo I, Roshanov P. et al; GUIDES expert panel. The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support. Implement Sci 2018; 13 (01) 86
• 35 Zheng WY, Van Dort B, Marcilly R. et al. A tool for evaluating medication alerting systems: development and initial assessment. JMIR Med Inform 2021; 9 (07) e24022
• 36 Kilbridge PM, Welebob EM, Classen DC. Development of the Leapfrog methodology for evaluating hospital implemented inpatient computerized physician order entry systems. Qual Saf Health Care 2006; 15 (02) 81-84
• 37 Zachariah M, Phansalkar S, Seidling HM. et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems–I-MeDeSA. J Am Med Inform Assoc 2011; 18 (suppl 1): i62-i72
• 38 Leiner DJ. SoSci Survey [Computer software]. 2019 . Accessed May 9, 2024 at: https://www.soscisurvey.de
• 39 Amkreutz J, Koch A, Buendgens L, Muehlfeld A, Trautwein C, Eisert A. Prevalence and nature of potential drug-drug interactions among kidney transplant patients in a German intensive care unit. Int J Clin Pharm 2017; 39 (05) 1128-1139
• 40 Amkreutz J, Koch A, Buendgens L, Trautwein C, Eisert A. Clinical decision support systems differ in their ability to identify clinically relevant drug interactions of immunosuppressants in kidney transplant patients. J Clin Pharm Ther 2017; 42 (03) 276-285
• 41 Seiberth S, Mannell H, Birkenmaier C. et al. Benefit of medication reviews by renal pharmacists in the setting of a computerized physician order entry system with clinical decision support. J Clin Pharm Ther 2022; 47 (10) 1531-1538
• 42 AiDKlinik. Accessed August 20, 2023 at: https://www.dosing-gmbh.de/produktloesungen/aidklinik-2/
• 43 Bittmann JA, Rein EK, Metzner M, Haefeli WE, Seidling HM. The acceptance of interruptive medication alerts in an electronic decision support system differs between different alert types. Methods Inf Med 2021; 60 (5-06): 180-184
• 44 Seidling HM, Schmitt SP, Bruckner T. et al. Patient-specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care 2010; 19 (05) e15
• 45 Pauly A, Wolf C, Busse M. et al. Evaluation of eight drug interaction databases commonly used in the German healthcare system. Eur J Hosp Pharm 2015; 22: 165-170
• 46 McCoy AB, Waitman LR, Lewis JB. et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc 2012; 19 (03) 346-352
• 47 Humphrey KE, Mirica M, Phansalkar S, Ozonoff A, Harper MB. Clinician perceptions of timing and presentation of drug-drug interaction alerts. Appl Clin Inform 2020; 11 (03) 487-496
• 48 Poly TN, Islam MM, Yang HC, Li YJ. Appropriateness of overridden alerts in computerized physician order entry: systematic review. JMIR Med Inform 2020; 8 (07) e15653
• 49 Bittmann JA, Haefeli WE, Seidling HM. Modulators influencing medication alert acceptance: an explorative review. Appl Clin Inform 2022; 13 (02) 468-485
• 50 Wright A, McEvoy DS, Aaron S. et al. Structured override reasons for drug-drug interaction alerts in electronic health records. J Am Med Inform Assoc 2019; 26 (10) 934-942
• 51 Coleman JJ, van der Sijs H, Haefeli WE. et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013; 13: 111
• 52 Simpao AF, Ahumada LM, Desai BR. et al. Optimization of drug-drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard. J Am Med Inform Assoc 2015; 22 (02) 361-369
• 53 Phansalkar S, van der Sijs H, Tucker AD. et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20 (03) 489-493
• 54 Wright A, Ash JS, Erickson JL. et al. A qualitative study of the activities performed by people involved in clinical decision support: recommended practices for success. J Am Med Inform Assoc 2014; 21 (03) 464-472
• 55 Zhang T, Gephart SM, Subbian V. et al. Barriers to adoption of tailored drug-drug interaction clinical decision support. Appl Clin Inform 2023; 14 (04) 779-788
• 56 Seidling HM, Stützle M, Hoppe-Tichy T. et al. Best practice strategies to safeguard drug prescribing and drug administration: an anthology of expert views and opinions. Int J Clin Pharm 2016; 38 (02) 362-373
• 57 Co Z, Holmgren AJ, Classen DC. et al. The development and piloting of the ambulatory electronic health record evaluation tool: lessons learned. Appl Clin Inform 2021; 12 (01) 153-163
• 58 Holmgren AJ, Kuznetsova M, Classen D, Bates DW. Assessing hospital electronic health record vendor performance across publicly reported quality measures. J Am Med Inform Assoc 2021; 28 (10) 2101-2107
• 59 Aarts J, Berg M. Same systems, different outcomes–comparing the implementation of computerized physician order entry in two Dutch hospitals. Methods Inf Med 2006; 45 (01) 53-61
• 60 By the 2023 American Geriatrics Society Beers Criteria Update Expert Panel. American Geriatrics Society 2023 updated AGS Beers Criteria for potentially inappropriate medication use in older adults. J Am Geriatr Soc 2023; 71 (07) 2052-2081
• 61 O'Mahony D, Cherubini A, Guiteras AR. et al. STOPP/START criteria for potentially inappropriate prescribing in older people: version 3. Eur Geriatr Med 2023; 14 (04) 625-632

Publikationsverlauf

Eingereicht: 15. Januar 2024

Angenommen: 01. April 2024

Artikel online veröffentlicht:
31. Juli 2024