Using Clinical Decision Support Systems to Decrease Intravenous Acetaminophen Use: Implementation and Lessons Learned

Keywords

clinical decision support system - quality improvement - electronic health record - pediatrics

Background and Significance

Clinical decision support systems (CDSS) are used to augment medical decision-making by providing targeted information. They may improve quality of care and reduce health care costs by increasing adherence to evidence-based knowledge and decreasing unwanted variations in care.[1] [2] [3] [4] Multiple studies have reported cost savings through various CDSS intervention types, including active and passive alerts and restrictive computerized provider order entry.[5] [6] [7] Although medication-related CDSS have myriad benefits when thoughtfully designed and implemented, they can also lead to unintended harm. Commonly cited risks include alert fatigue, fragmented workflows, and high outset and ongoing maintenance costs.[8] This can be particularly true with CDSS that use hard-stops, in which the user is prevented from taking an action altogether.[9]

Intravenous (IV) acetaminophen has been an enticing target for CDSS intervention in recent years. Approved by the U.S. Food and Drug Administration in 2010, it can cost hundreds of times more than the oral tablet formulation.[10] [11] [12] Acetaminophen is commonly used for its antipyretic and analgesic properties and strong safety profile.[13] [14] Although the IV route may be advantageous when oral or rectal routes are contraindicated, it is unclear if IV acetaminophen is more efficacious in general.[15] [16] In this study, we describe the implementation of a hard-stop CDSS, which was chosen as an intervention within a quality improvement initiative that sought to promote appropriate use of IV acetaminophen, with the primary aim to decrease hospital-wide IV acetaminophen administrations per 1,000 patient days by 20% over a 12-month period.

Methods

Lucile Packard Children's Hospital (LPCH) is an academic children's hospital in Palo Alto, California with 361 inpatient beds. It is affiliated with Stanford School of Medicine, with ordering providers including trainees, advanced practice providers, and attending physicians. In 2016, the Pharmacy and Therapeutics Committee analyzed IV acetaminophen usage and determined that it represented among the highest inpatient medication expenditures within the hospital. LPCH's utilization rates were significantly higher compared with other children's hospitals across the United States.

In response, we formed a multidisciplinary workgroup with the aim of reducing the inappropriate use and overall expenditure of IV acetaminophen. The workgroup consisted of pharmacists, ordering providers, clinical informaticians, data analysts, and performance improvement consultants. Inappropriate use of IV acetaminophen was defined as ordering the IV formulation when there was no contraindication to enteral routes. Current state analysis revealed that 60% of IV acetaminophen orders may have been inappropriate, with 41% of IV acetaminophen orders prescribed for patients without nil per oral (NPO) orders and 19% of IV acetaminophen orders prescribed for patients with NPO orders that had exceptions for medications. The target of reducing IV acetaminophen administration by 20% was chosen as it represented a 33% decrease in potentially inappropriate IV acetaminophen orders compared with the current state, which the workgroup agreed would be an attainable goal. We used quality improvement methodology with the Plan-Do-Study-Act (PDSA) model for improvement to reach our aim. SQUIRE 2.0 guidelines were used as a framework to report this quality improvement initiative.[17]

The workgroup initiated a series of electronic health record (EHR)-based interventions aimed at decreasing IV acetaminophen administration. One intervention included an interruptive alert within the EHR (Epic Systems, Inc., Verona, Wisconsin, United States), which prompted providers ordering IV acetaminophen to consider ordering an enteral formulation when the patient had an active diet order. Another intervention defaulted the duration of each new IV acetaminophen order to 24 hours, although this could be overridden by the ordering provider. A third intervention generated an acetaminophen order set that included both IV and oral formulations along with guidance to avoid the IV option if the patient was feeding enterally. All three interventions were unsuccessful in decreasing the number of IV acetaminophen administrations. A separate non-EHR-based intervention was implemented in the cardiovascular intensive care unit (CVICU) in August 2020, which involved a pharmacist joining rounds and specifically recommending that ordering providers change from IV to enteral acetaminophen when clinically indicated.

In response to the lack of success in the three EHR-based interventions, we implemented a hard-stop CDSS that set a mandatory expiry time of 24 hours for all IV acetaminophen orders. Thus, continuing IV acetaminophen after 24 hours would require a new order from the ordering provider. The intent was for IV acetaminophen orders to be reviewed every day to prompt more opportunity for discontinuation or transition to enteral formulations as appropriate.

This hard-stop CDSS went live on April 5, 2021 throughout all inpatient units, accompanied by education to ordering providers and nurses. Our primary outcome measure was the number of IV acetaminophen administrations per 1,000 patient days, which was measured pre- and postimplementation. This outcome measure was chosen as a standard unit of measurement because it calculates the rate IV acetaminophen administrations, which accounts for variation in patient volume.

Process measures included the number of IV acetaminophen orders per 1,000 patient days that were placed. This was chosen as a process measure as we anticipated that a reduction in IV acetaminophen orders would be an important step for reducing total IV acetaminophen administrations.

Recognizing the potential pitfalls of hard-stop CDSS, including possible delays in care and provider dissatisfaction, we used these as our balancing measures, which were measured through a survey sent to a convenience sample of inpatient nurses and ordering providers postimplementation.

Control charts (also called Shewhart charts or statistical process control charts) were used for analysis, which is commonly used in improvement science methodology. Control charts are often used to distinguish between variation in a system caused by common causes from those due to special causes, and they are able to detect process changes and trends earlier than classical statistical methods.[18] Common causes are those inherent in the system, whereas special causes are causes of variation that are not part of the system, such as a specific intervention. There are several rules to identify special cause variation, but some of the most common rules include (1) shifts, when six or more consecutive points are all above or below the median value, (2) trends, when five or more consecutive points all go in the same direction, and (3) astronomical points, where a data point is outside of the upper and lower control limits. Signals of special cause variation are important to recognize, as they may represent that an intervention has led to improvement.[19] [20]

This study was Institutional Review Board exempt as it did not meet the criteria for research by our Research Compliance Office.

Results

Outcome Measure: Changes in Intravenous Acetaminophen Administration per 1,000 Patient Days

The number of IV acetaminophen administrations per 1,000 patient days was measured from January 1, 2020 to October 31, 2022. When examining absolute number of IV acetaminophen administrations, the CVICU was the largest utilizer, with 46% of total hospital administrations ([Fig. 1]). However, when adjusting for number of patient days, the pediatric intensive care unit (PICU) was the highest utilizer of IV acetaminophen per 1,000 patient days, followed by the CVICU and the acute care units.

Fig. 1 Pareto chart displaying percentage of total IV acetaminophen administrations per unit. All other units include neonatal intensive care unit and short stay unit. CVICU, cardiovascular intensive care unit; IV, intravenous; PICU, pediatric intensive care unit.

Hospital-wide administration of IV acetaminophen was 289 times per 1,000 patient days from January 1, 2020 to December 31, 2021 ([Fig. 2]). There was a downward trend, defined as five or more consecutive points all going down, from August 2020 to January 2021 ([Fig. 2]), and this correlates with a shift, defined as six or more consecutive points below the median, in the CVICU during the same period ([Fig. 3]).[19] When assessing IV acetaminophen administration on acute care units specifically, this did not reveal a similar trend or shift ([Fig. 4]). The PICU had an average of 541 IV acetaminophen administrations per 1,000 patient days from January 1, 2020 to October 31, 2022 ([Fig. 5]). The hard-stop CDSS was implemented on April 5, 2021, and this intervention was not associated with special cause variation in the CVICU, acute care units, PICU, or hospital wide ([Figs. 2], [3], [4], [5]). Similarly, there was no special cause variation after the hard-stop CDSS was removed in April 2022. There was no change in the percentage of IV acetaminophen administrations based on NPO order status after hard-stop CDSS implementation and removal ([Fig. 6]).

Fig. 2 Control chart of IV acetaminophen administrations per 1,000 patient days throughout the hospital. CL, center line; IV, intravenous; LCL, lower control limit; UCL, upper control limit.

Fig. 3 Control chart of IV acetaminophen administrations per 1,000 patient days in the CVICU. CDSS, clinical decision support systems; CVICU, cardiovascular intensive care unit; IV, intravenous; LCL, lower control limit; UCL, upper control limit.

Fig. 4 Control chart of IV acetaminophen administrations per 1,000 patient days in acute care units (inpatient units exclusive of PICU and CVICU). CDSS, clinical decision support systems; IV, intravenous; LCL, lower control limit; UCL, upper control limit.

Fig. 5 Control chart of IV acetaminophen administrations per 1,000 patient days in the PICU. CDSS, clinical decision support systems; CL, center line; IV, intravenous; LCL, lower control limit; PICU, pediatric intensive care unit; UCL, upper control limit.

Fig. 6 Line graph of the percentage of hospital-wide IV acetaminophen administrations based on nil per oral (NPO) status. CDSS, clinical decision support systems; IV, intravenous;.

Process Measure: Changes in Intravenous Acetaminophen Orders per 1,000 Patient Days

The number of orders placed for IV acetaminophen per 1,000 patient days was measured from January 1, 2020 to September 30, 2022. Hospital wide, there were an average of 140 IV acetaminophen orders per 1,000 patient days ([Fig. 7]). When stratified based on unit, the PICU had the highest number of IV acetaminophen orders per 1,000 patient days, with an average of 170, compared with acute care units (average 54) and the CVICU (average 77) ([Figs. 8] [9] [10]). No special cause variation was seen with the implementation or removal of the hard-stop CDSS in the CVICU, acute care units, PICU, and hospital wide.

Fig. 7 Control chart of IV acetaminophen orders placed per 1,000 patient days throughout the hospital. CDSS, clinical decision support systems; IV, intravenous; LCL, lower control limit; UCL, upper control limit.

Fig. 8 Control chart of IV acetaminophen orders placed per 1,000 patient days in the CVICU. CDSS, clinical decision support systems; IV, intravenous; LCL, lower control limit; UCL, upper control limit.

Fig. 9 Control chart of IV acetaminophen orders placed per 1,000 patient days on Acute Care Units. CDSS, clinical decision support systems; CVICU, cardiovascular intensive care unit; IV, intravenous; LCL, lower control limit; UCL, upper control limit.

Fig. 10 Control chart of IV acetaminophen orders placed per 1,000 patient days in the PICU. CDSS, clinical decision support systems; IV, intravenous; LCL, lower control limit; PICU, pediatric intensive care unit; UCL, upper control limit.

Survey Results

A total of 88 participants completed the survey. Of these, 80% (70/88) were nurses and 20% (18/88) were ordering providers. Nearly half (45%, 40/88) of respondents reported negative issues with the 24-hour hard-stop CDSS (47% (33/70) of nurses, 39% (7/18) of ordering providers), with the majority stating that this affected patient care (39/40). The most frequently cited examples involved delays in receiving IV acetaminophen for patients in pain. Moreover, 80% (32/40) of respondents who experienced negative issues reported that the CDSS affected their efficiency. For example, 52% (46/88) of nurses reported contacting an ordering provider at least once in the past month to reorder IV acetaminophen. Accordingly, 56% (10/18) of ordering providers reported being contacted by a nurse to reorder IV acetaminophen.

Discussion

We provide an example of a CDSS that had minimal effect on the targeted clinician behavior while being burdensome to clinical staff and thus was appropriately removed from the EHR. Our intervention did not lead to a reduction in the number of IV acetaminophen administrations per 1,000 patient days and was not a special cause of variation. Similarly, CDSS removal was not associated with special cause variation. It was associated with low provider acceptability and potential negative impacts on patient care. The lack of decrease in IV acetaminophen administrations suggest that it was often reordered once the existing 24-hour order expired, which would require nurses to contact the ordering provider to reorder IV acetaminophen, leading to fragmentation in workflows, interruptions, and additional workload for nurses, ordering providers, and pharmacists.

Notably, there was a reduction in hospital-wide IV acetaminophen administration from August 2020 to January 2021, which was also present in the CVICU but not on acute care units. This special cause variation is likely related to a local CVICU intervention, where the CVICU pharmacist reviewed all IV acetaminophen orders on rounds and requested providers to transition to enteral routes of administration if clinically indicated. Of note, this CVICU pharmacy intervention was not associated with a reduction in our process measure, as no special cause variation was seen in IV acetaminophen orders per 1,000 patient days. This suggests that the reduction in IV acetaminophen administrations was directly related to the CVICU pharmacist cancelling these orders, rather than ordering providers changing their behavior and ordering IV acetaminophen less frequently. This highlights the notion that technology-based solutions, particularly CDSS, are often not a “magic bullet” and must be paired with strategies that focus on changing individual behavior to have a sustained effect.

Although this CDSS was ultimately removed, aspects of our implementation strategy were executed well and offer valuable lessons. Firstly, we implemented the least burdensome strategies first, with interruptive interventions attempted only when it was clear that the prior interventions were ineffective. Active CDSS, including interruptive and hard-stop alerts, carry higher risk as they mandate that the provider change their workflow. This was exemplified in a randomized trial that used hard-stop alerts to reduce the concomitant ordering of two medications that had drug interactions when given together. The study was terminated early as there were delays in appropriate treatment being administered due to the hard-stop CDSS, which highlights the often unforeseen consequences that CDSS may have.[21] In addition, CDSS have been associated with end user frustration and burnout, which may be exacerbated by hard-stops.[9] [22] In most cases, interventions should be implemented in a stepwise fashion, starting with the least invasive strategy to minimize the risk of unintended consequences associated with active CDSS.[9]

In keeping with the PDSA model for improvement, our workgroup used time-limited pilot trials for all implemented CDSS and met on a regular basis to monitor effectiveness, gather end user feedback, and determine if the CDSS should be continued. Leveraging end user feedback allowed us to monitor for unintended consequences, understand barriers to use, and make iterative changes to increase adherence. Furthermore, while not performed in this study, balancing measures can be measured objectively by collecting data from the EHR; for example, if a CDSS is frequently dismissed by the end user, this signals poor acceptability. This approach is supported by multiple CDSS frameworks, including the Ten Commandments for Effective Clinical Decision Support and the GUIDES checklist for CDSS implementation, which recommend that end user feedback be gathered throughout the process of implementation.[23] [24] A formal evaluation process allowed us to minimize dissatisfaction from providers by promptly discontinuing the CDSS when we learned of its unintended consequences.

As we consider how this CDSS was developed and implemented, we also recognize opportunities for future improvement. CDSS failure can often be anticipated by clinical informaticians based on CDSS design limitations and existing cultural practices within the clinical area. In our example, the CDSS had several features that contributed to its failure. In terms of CDSS design, it forced providers to stop an action (i.e., ordering IV acetaminophen) without offering an alternative option, and it added to the end user's workflow by requiring repeated ordering of IV acetaminophen every 24 hours. The purpose of the CDSS was largely to reduce costs and not improve patient outcomes, and this economic driver was not aligned with that of the end users who were focused on providing direct patient care. As the price of IV acetaminophen decreased with time, there was less impetus to continue this project because our aim was not tied to clinical outcomes. Finally, while the workgroup was multidisciplinary and had preconceptions about why IV acetaminophen was being inappropriately used, a formal analysis was not performed. As improvement methodology focuses on understanding the system prior to intervening, future projects should seek to understand the reasons that influence ordering provider decisions prior to and during implementation. This would have allowed us to better tailor our interventions to address the specific causes. If we critically considered the risk of CDSS failure prior to development and implementation, we may have more readily recognized the futility of pursuing this CDSS.

Significant resources were invested and include time spent by personnel to develop, implement, and maintain the CDSS, the opportunity cost of not pursuing other projects, and the difficult-to-quantify cost of provider dissatisfaction. While CDSS has been emphasized as a solution to rising health care costs, the evidence for its cost-effectiveness and cost–benefit is mixed.[25] [26] In our case, performing a cost–benefit analysis prior to pursuing this CDSS may have determined that the potential cost savings from reduced IV acetaminophen administration would not surpass the resource costs associated with development, implementation, and monitoring.

Our work has important limitations. While our outcome measure accounts for variability in patient volume by measuring IV acetaminophen administrations per 1,000 patient days, it does not control for variability in case mix index. This is a potential confounder, particularly in pediatrics where the case mix is highly variable based on seasonality and with the COVID-19 pandemic, which also had repercussions on case mix index. Despite this, the CDSS was implemented for 12 months prior to removal, and there was no significant change in IV acetaminophen administrations throughout those 12 months. This CDSS was implemented at a single tertiary, pediatric hospital within the inpatient units. Thus, the lessons from this case may not be fully generalizable to other settings. In addition, our survey data used convenience sampling of nurses and ordering providers, which may have resulted in selection bias. The survey's primary purpose was to understand provider and nursing acceptability of the interruptive CDSS, and it did not ascertain if there was a clinical reason for ordering IV acetaminophen instead of the enteral formulation. Analyzing the basis for IV acetaminophen orders would have better allowed us to target interventions that address root causes.

This case report highlights areas of future direction to improve effective CDSS implementation. Currently, features that predict CDSS success or failure are not well elucidated, with few studies reporting on system design and implementation features.[27] Creation of an evidence-based tool for predicting CDSS success would help clinical informaticians better allocate resources toward CDSS that have higher likelihood of success. Being able to risk stratify CDSS may also prompt teams to monitor more closely already-implemented CDSS that are deemed high risk for failure. Particularly for CDSS that aims to decrease utilization, development of a cost–benefit analysis tool would allow clinical informaticians to quantify the potential cost savings of decreased utilization compared with the costs of CDSS development, implementation, and monitoring. These models could help bridge the gap between clinical informatics teams and health system stakeholders requesting CDSS, and it may impartially demonstrate when a requested CDSS is at high risk for failure and allow for utilization of health care resources toward interventions that are more likely to succeed.

Conclusion

CDSS have the potential to improve clinical practice and may reduce costs when effectively implemented.[25] [26] [28] This case report highlights the importance of monitoring and gathering feedback after the implementation of any CDSS, particularly around additional provider burden and workload. Teams should be prepared to modify or remove the CDSS if end user acceptability is low, particularly when the primary aim of the CDSS is not definitively achieved. Models that predict CDSS failure and cost–benefit may be helpful in allocating resources to CDSS that have high likelihood of success and greater economic benefit.

Clinical Relevance Statement

CDSS are increasingly used to improve quality of care and decrease cost. However, they can lead to unintended harm by causing alert fatigue, fragmented workflows, and frustration from the end user. All CDSS implementations should have a system for monitoring efficacy and gathering feedback from end users, and teams should be prepared to modify or remove the CDSS if end user acceptability is low or the desired effect is not achieved.

Multiple-Choice Questions

1. What is a commonly cited risk specific to implementing hard-stop CDSS?

   • High cost of development

   • Changes to end user workflow

   • Lack of efficacy

   • Low rates of adoption

   Correct Answer: The correct answer is option b. Hard-stop CDSS is a type of active CDSS that requires a certain behavior from the end user and requires the end user to change their existing workflow. The other options are risks when implementing all types of CDSS, but these are not specific to hard-stop CDSS.

2. Which is a framework that can be used when developing and implementing effective CDSS?

   • The GUIDES checklist

   • Five Commandments for Effective Clinical Decision Support

   • The Technology Acceptance Model

   • The DeLone and McLean Information Systems Success Model

   Correct Answer: The correct answer is option a. The GUIDES checklist consists of 16 factors that assists professionals when implementing guidelines with CDSS. Option b is incorrect; Ten Commandments for Effective Clinical Decision Support is a common framework used when developing effective CDSS. Options c and d are incorrect as these are not frameworks related specifically to CDSS.

• References

• 1 Das M, Eichner J. . Challenges and Barriers to Clinical Decision Support (CDS) Design and Implementation Experienced in the Agency for Healthcare Research and Quality CDS Demonstrations (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016.) AHRQ Publication No. 10-0064-EF. Rockville, MD: Agency for Healthcare Research and Quality. March 2010.
• 2 Kharbanda EO, Asche SE, Sinaiko AR. et al. Clinical decision support for recognition and management of hypertension: a randomized trial. Pediatrics 2018; 141 (02) e20172954
• 3 Prgomet M, Li L, Niazkhani Z, Georgiou A, Westbrook JI. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. J Am Med Inform Assoc 2017; 24 (02) 413-422
• 4 Hayatghaibi SE, Sammer MBK, Varghese V, Seghers VJ, Sher AC. Prospective cost implications with a clinical decision support system for pediatric emergency head computed tomography. Pediatr Radiol 2021; 51 (13) 2561-2567
• 5 Lewkowicz D, Wohlbrandt A, Boettinger E. Economic impact of clinical decision support interventions based on electronic health records. BMC Health Serv Res 2020; 20 (01) 871
• 6 MacMillan TE, Gudgeon P, Yip PM, Cavalcanti RB. Reduction in unnecessary red blood cell folate testing by restricting computerized physician order entry in the electronic health record. Am J Med 2018; 131 (08) 939-944
• 7 Chin KK, Hom J, Tan M. et al. Effect of electronic clinical decision support on 25(OH) vitamin D testing. J Gen Intern Med 2019; 34 (09) 1697-1699
• 8 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3 (01) 17
• 9 Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018; 25 (11) 1556-1566
• 10 Sweeney J. IV vs. oral acetaminophen for children: weighing cost against need. Pharmacy Today. 2019; 25 (06) 4
• 11 Bourgeois FT, Graham DA, Kesselheim AS, Randolph AG. Cost implications of escalating intravenous acetaminophen use in children. JAMA Pediatr 2019; 173 (05) 489-491
• 12 Nguyen LP, Nguyen L, Austin JP. A quality improvement initiative to decrease inappropriate intravenous acetaminophen use at an academic medical center. Hosp Pharm 2020; 55 (04) 253-260
• 13 Bertolini A, Ferrari A, Ottani A, Guerzoni S, Tacchi R, Leone S. Paracetamol: new vistas of an old drug. CNS Drug Rev 2006; 12 (3-4): 250-275
• 14 Jahr JS, Lee VK. Intravenous acetaminophen. Anesthesiol Clin 2010; 28 (04) 619-645
• 15 Tompkins DM, DiPasquale A, Segovia M, Cohn SM. Review of intravenous acetaminophen for analgesia in the postoperative setting. Am Surg 2021; 87 (11) 1809-1822
• 16 Wasserman I, Poeran J, Zubizarreta N. et al. Impact of intravenous acetaminophen on perioperative opioid utilization and outcomes in open colectomies: a claims database analysis. Anesthesiology 2018; 129 (01) 77-88
• 17 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf 2016; 25 (12) 986-992
• 18 Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003; 12 (06) 458-464
• 19 Langley G, Moen R, Nolan K, Nolan T, Norman C, Provost L. The Improvement Guide. 2nd ed. Jossey-Bass; 2009
• 20 Carroll AR, Johnson DP. Know it when you see it: identifying and using special cause variation for quality improvement. Hosp Pediatr 2020; 10 (11) e8-e10
• 21 Strom BL, Schinnar R, Aberra F. et al. Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010; 170 (17) 1578-1583
• 22 Jankovic I, Chen JH. Clinical decision support and implications for the clinician burnout crisis. Yearb Med Inform 2020; 29 (01) 145-154
• 23 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 24 Van de Velde S, Kunnamo I, Roshanov P. et al; GUIDES expert panel. The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support. Implement Sci 2018; 13 (01) 86
• 25 Jacob V, Thota AB, Chattopadhyay SK. et al. Cost and economic benefit of clinical decision support systems for cardiovascular disease prevention: a community guide systematic review. J Am Med Inform Assoc 2017; 24 (03) 669-676
• 26 Bright TJ, Wong A, Dhurjati R. et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (01) 29-43
• 27 Mollon B, Chong Jr J, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med Inform Decis Mak 2009; 9 (01) 11
• 28 Kwan JL, Lo L, Ferguson J. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 370: m3216