The Costs and Benefits of Clinical Decision Support for Radiology Appropriate Use Criteria: A Retrospective Observational Study

Abstract

Background

Appropriate Use Criteria Clinical Decision Support (AUC CDS) was legislatively mandated in the United States in 2014, and multiple CDS vendors were designated as qualified Clinical Decision Support Mechanisms by the Centers for Medicare and Medicaid Services. Little is known about the costs and benefits of these systems in real-world settings.

Objectives

We evaluated the effectiveness of an AUC CDS system and the time costs it imposes on clinicians at an academic medical center.

Methods

Our U.S. academic medical center's enterprise data warehouse was queried for AUC CDS alert events and timestamps occurring between July 1, 2021, and June 30, 2022. We calculated the percentage of altered orders and alert-related timespans, and used these to calculate CDS positive predictive value (PPV), time costs, and the cost–benefit ratio of minutes of provider time per altered order. Based on the medical literature and expert opinion on well-performing CDS, we hypothesized a CDS PPV of 8%.

Results

Overall PPV was 1%, leading us to reject our hypothesis that our CDS was well-performing (p < 0.001). Median time costs per alert were high (12 seconds load time, 2 seconds dwell time), yielding a CDS cost–benefit ratio of 38 provider minutes per altered order.

Conclusion

Despite using one of three market-leading AUC CDS tools, our CDS demonstrated long load times, short dwell times, and low PPV. Provider attention is not free—policymakers should consider both CDS effectiveness and costs (including time costs) when designing AUC policy.

Keywords

clinical decision support - electronic health records and systems - regulatory and policy issues - radiology - alerting

• References

• 1 Smith-Bindman R, Kwan ML, Marlow EC. et al. Trends in use of medical imaging in US health care systems and in Ontario, Canada, 2000-2016. JAMA 2019; 322 (09) 843-856
• 2 Smith-Bindman R, Miglioretti DL, Larson EB. Rising use of diagnostic medical imaging in a large integrated health system. Health Aff (Millwood) 2008; 27 (06) 1491-1502
• 3 Kocher KE, Meurer WJ, Fazel R, Scott PA, Krumholz HM, Nallamothu BK. National trends in use of computed tomography in the emergency department. Ann Emerg Med 2011; 58 (05) 452-62.e3
• 4 Smith-Bindman R, Miglioretti DL, Johnson E. et al. Use of diagnostic imaging studies and associated radiation exposure for patients enrolled in large integrated health care systems, 1996-2010. JAMA 2012; 307 (22) 2400-2409
• 5 Sodickson A, Baeyens PF, Andriole KP. et al. Recurrent CT, cumulative radiation exposure, and associated radiation-induced cancer risks from CT of adults. Radiology 2009; 251 (01) 175-184
• 6 Lehnert BE, Bree RL. Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an academic medical center: how critical is the need for improved decision support?. J Am Coll Radiol 2010; 7 (03) 192-197
• 7 Waheed S, Tahir MJ, Ullah I. et al. The impact of dependence on advanced imaging techniques on the current radiology practice. Ann Med Surg (Lond) 2022; 78: 103708
• 8 American College of Radiology. ACR Appropriateness Criteria. Accessed November 28, 2023 at: https://www.acr.org/Clinical-Resources/ACR-Appropriateness-Criteria
• 9 Kurth DA, Karmazyn BK, Waldrip CA, Chatfield M, Lockhart ME. ACR Appropriateness Criteria® Methodology. J Am Coll Radiol 2021; 18 (11S): S240-S250
• 10 Bautista AB, Burgos A, Nickel BJ, Yoon JJ, Tilara AA, Amorosa JK. American College of Radiology Appropriateness. Do clinicians use the American College of Radiology Appropriateness criteria in the management of their patients?. AJR Am J Roentgenol 2009; 192 (06) 1581-1585
• 11 Congress.Gov. H.R. 4302 - Protecting Access to Medicare Act of 2014. 2014. Accessed November 28, 2023 at: https://www.congress.gov/bill/113th-congress/house-bill/4302/text
• 12 Medicare Program; Revisions to Payment Policies under the Physician Fee Schedule and Other Revisions to Part B for CY 2018. Medicare Shared Savings Program Requirements; and Medicare Diabetes Prevention Program. Accessed May 15, 2025 at: https://www.govinfo.gov/content/pkg/FR-2017-11-15/pdf/2017-23953.pdf
• 13 Appropriate use criteria program. CMS.gov., September 10, 2024. Accessed July 16, 2025 at: https://www.cms.gov/medicare/quality/appropriate-use-criteria-program
• 14 American College of Radiology. ACR Backs ROOT Act to Reignite AUC Program Implementation. 2025. Accessed May 15, 2025 at: https://www.acr.org/News-and-Publications/Media-Center/2025/acr-backs-root-act
• 15 Zygmont ME, Ikuta I, Nguyen XV, Frigini LAR, Segovis C, Naeger DM. Clinical decision support: Impact on appropriate imaging utilization. Acad Radiol 2023; 30 (07) 1433-1440
• 16 Liu VX, Bates DW, Wiens J, Shah NH. The number needed to benefit: estimating the value of predictive analytics in healthcare. J Am Med Inform Assoc 2019; 26 (12) 1655-1659
• 17 Cánovas-Segura B, Morales A, Juarez JM, Campos M. Meaningful time-related aspects of alerts in clinical decision support systems. A unified framework. J Biomed Inform 2023; 143: 104397
• 18 Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of decision support in computerized provider order entry: Systematic literature review. JMIR Med Inform 2018; 6 (01) e3
• 19 Payne TH, Desai BR. Examination of medication clinical decision support using Bayes' theorem. Am J Health Syst Pharm 2016; 73 (22) 1876-1878
• 20 Carspecken CW, Sharek PJ, Longhurst C, Pageler NM. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics 2013; 131 (06) e1970-e1973
• 21 Gregory ME, Russo E, Singh H. Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8 (03) 686-697
• 22 Rousseau JF, Ip IK, Raja AS. et al. Can automated retrieval of data from emergency department physician notes enhance the imaging order entry process?. Appl Clin Inform 2019; 10 (02) 189-198
• 23 Brunner MC, Sheehan SE, Yanke EM. et al. Joint design with providers of clinical decision support for value-based advanced shoulder imaging. Appl Clin Inform 2020; 11 (01) 142-152
• 24 Bates DW, Kuperman GJ, Wang S. et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 25 Li C, Parpia C, Sriharan A, Keefe DT. Electronic medical record-related burnout in healthcare providers: a scoping review of outcomes and interventions. BMJ Open 2022; 12 (08) e060865
• 26 Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014; 12 (06) 573-576