A Measurement Science Framework to Optimize CDS for Opioid Use Disorder Treatment in the ED

Abstract

Objectives

In the emergency department-initiated buprenorphine for opioid use disorder (EMBED) trial, a clinical decision support (CDS) tool had no effect on rates of buprenorphine initiation in emergency department (ED) patients with opioid use disorder. The Agency for Healthcare Research and Quality (AHRQ) recently released a CDS Performance Measure Inventory to guide data-driven CDS development and evaluation. Through partner co-design, we tailored AHRQ inventory measures to evaluate EMBED CDS performance and drive improvements.

Methods

Relevant AHRQ inventory measures were selected and adapted using a partner co-design approach grounded in consensus methodology, with three iterative, multidisciplinary partner working group sessions involving stakeholders from various roles and institutions; meetings were followed by postmeeting surveys. The co-design process was divided into conceptualization, specification, and evaluation phases building on the Centers for Medicare and Medicaid Services' measure life cycle framework. Final measures were evaluated in three EDs in a single health system from January 1, 2023, to December 31, 2024.

Results

The partner working group included 25 members. During conceptualization, 13 initial candidate metrics were narrowed to 6 priority categories. These were further specified and validated as the following measures, presented with preliminary values based on the use of the current (i.e., preoptimization) EMBED CDS: eligible encounters with CDS engagement, 5.0% (95% confidence interval: 4.3–5.8%); teamwork on ED initiation of buprenorphine, 39.9% (32.5–47.3%); proportion of eligible users who used EMBED, 58.3% (50.9–65.8%); time spent on EMBED, 29.0 seconds (20.4–37.7 seconds); proportion of buprenorphine orders placed through EMBED, 6.5% (3.4–9.6%); and task completion, 13.8% (8.9–18.7%) for buprenorphine order/prescription.

Conclusion

A measurement science framework informed by partner co-design was a feasible approach to develop measures to guide CDS improvement. Subsequent research could adapt this approach to evaluate other CDS applications.

Keywords

emergency medicine - addiction - workflow - measurement and observation - electronic health records and systems

• References

• 1 Samples H, Nowels MA, Williams AR, Olfson M, Crystal S. Buprenorphine after nonfatal opioid overdose: reduced mortality risk in medicare disability beneficiaries. Am J Prev Med 2023; 65 (01) 19-29
• 2 Heimer R, Black AC, Lin H. et al. Receipt of opioid use disorder treatments prior to fatal overdoses and comparison to no treatment in Connecticut, 2016-17. Drug Alcohol Depend 2024; 254: 111040
• 3 Chua KP, Nguyen TD, Zhang J, Conti RM, Lagisetty P, Bohnert AS. Trends in buprenorphine initiation and retention in the United States, 2016-2022. JAMA 2023; 329 (16) 1402-1404
• 4 Larochelle MR, Bernson D, Land T. et al. Medication for opioid use disorder after nonfatal opioid overdose and association with mortality: a cohort study. Ann Intern Med 2018; 169 (03) 137-145
• 5 Langabeer JR, Stotts AL, Bobrow BJ. et al. Prevalence and charges of opioid-related visits to U.S. emergency departments. Drug Alcohol Depend 2021; 221 (108568): 108568
• 6 D'Onofrio G, O'Connor PG, Pantalon MV. et al. Emergency department-initiated buprenorphine/naloxone treatment for opioid dependence: a randomized clinical trial. JAMA 2015; 313 (16) 1636-1644
• 7 Ray JM, Ahmed OM, Solad Y. et al. Computerized clinical decision support system for emergency department-initiated buprenorphine for opioid use disorder: user-centered design. JMIR Hum Factors 2019; 6 (01) e13121
• 8 Melnick ER, Holland WC, Ahmed OM. et al. An integrated web application for decision support and automation of EHR workflow: a case study of current challenges to standards-based messaging and scalability from the EMBED trial. JAMIA Open 2019; 2 (04) 434-439
• 9 Kwan JL, Lo L, Ferguson J. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ 2020; 370: m3216
• 10 Melnick ER, Nath B, Dziura JD. et al. User centered clinical decision support to implement initiation of buprenorphine for opioid use disorder in the emergency department: EMBED pragmatic cluster randomized controlled trial. BMJ 2022; 377: e069271
• 11 Melnick ER, Sinsky CA, Krumholz HM. Implementing measurement science for electronic health record use. JAMA 2021; 325 (21) 2149-2150
• 12 Kannampallil T, Adler-Milstein J. Using electronic health record audit log data for research: insights from early efforts. J Am Med Inform Assoc 2022; 30 (01) 167-171
• 13 Rule A, Kannampallil T, Hribar MR. et al. Guidance for reporting analyses of metadata on electronic health record use. J Am Med Inform Assoc 2024; 31 (03) 784-789
• 14 Levy DR, Moy AJ, Apathy N. et al. Identifying and addressing barriers to implementing core electronic health record use metrics for ambulatory care: virtual consensus conference proceedings. Appl Clin Inform 2023; 14 (05) 944-950
• 15 Adler-Milstein J, Adelman JS, Tai-Seale M, Patel VL, Dymek C. EHR audit logs: a new goldmine for health services research?. J Biomed Inform 2020; 101 (103343): 103343
• 16 Dullabh PM, Heaney-Huls K, Jiménez F. et al. CDSiC Scaling, Measurement, and Dissemination of CDS Workgroup. Scaling, Measurement, and Dissemination of CDS Workgroup: PC CDS Performance Measurement Inventory User Guide. Prepared under Contract No. 75Q80120D00018. AHRQ Publication No. 23–0073; 2023
• 17 Clarke D, Jones F, Harris R, Robert G. Collaborative Rehabilitation Environments in Acute Stroke (CREATE) team. What outcomes are associated with developing and implementing co-produced interventions in acute healthcare settings? A rapid evidence synthesis. BMJ Open 2017; 7 (07) e014650
• 18 Ward ME, De Brún A, Beirne D. et al. Using co-design to develop a collective leadership intervention for healthcare teams to improve safety culture. Int J Environ Res Public Health 2018; 15 (06) 1182
• 19 Cooper L, Ahmed S, Bibeau C. et al. Impact of patient partner co-design on survey development in primary care research: case study. Can Fam Physician 2022; 68 (03) 235-236
• 20 Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007; 32 (05) S112-S118
• 21 Iscoe MS, Diniz Hooper C, Levy DR. et al. Adaptive decision support for addiction treatment to implement initiation of buprenorphine for opioid use disorder in the emergency department: protocol for the ADAPT multiphase optimization strategy trial. BMJ Open 2025; 15 (02) e098072
• 22 Blueprint Measure Lifecycle Overview. . Accessed December 20, 2024 at: https://mmshub.cms.gov/blueprint-measure-lifecycle-overview
• 23 Jandhyala R. Delphi, non-RAND modified Delphi, RAND/UCLA appropriateness method and a novel group awareness and consensus methodology for consensus measurement: a systematic literature review. Curr Med Res Opin 2020; 36 (11) 1873-1887
• 24 Nesbitt K, Beleigoli A, Du H, Tirimacco R, Clark RA. User experience (UX) design as a co-design methodology: lessons learned during the development of a web-based portal for cardiac rehabilitation. Eur J Cardiovasc Nurs 2022; 21 (02) 178-183
• 25 Williams C, Woods L, Stott A, Duff J. Codesigning an E-health intervention for surgery preparation and recovery. Comput Inform Nurs 2024; 42 (09) 655-664
• 26 McGlynn EA, Adams JL. What makes a good quality measure?. JAMA 2014; 312 (15) 1517-1518
• 27 Chartash D, Paek H, Dziura JD. et al. Identifying opioid use disorder in the emergency department: multi-system electronic health record-based computable phenotype derivation and validation study. JMIR Med Inform 2019; 7 (04) e15794
• 28 Griffin AC, Perez T, Avoundjian T, Becker W, Midboe AM. A tool to identify and engage patients on risky opioid regimens. Appl Clin Inform 2023; 14 (05) 1018-1026
• 29 Jones NW, Song SL, Thomasian N, Samuels EA, Ranney ML. Behavioral health decision support systems and user interface design in the emergency department. Appl Clin Inform 2023; 14 (04) 705-713
• 30 Allen KS, Danielson EC, Downs SM. et al. Evaluating a prototype clinical decision support tool for chronic pain treatment in primary care. Appl Clin Inform 2022; 13 (03) 602-611
• 31 Brown T, Zelch B, Lee JY. et al. A qualitative description of clinician free-text rationales entered within accountable justification interventions. Appl Clin Inform 2022; 13 (04) 820-827
• 32 Salwei ME, Hoonakker PLT, Pulia M, Wiegmann D, Patterson BW, Carayon P. Post-implementation usability evaluation of a human factors-based clinical decision support for pulmonary embolism (PE) diagnosis (Dx): PE Dx Study Part 1. Hum Factors Healthc 2023; 4 (100056): 100056
• 33 Carayon P, Hoonakker P, Hundt AS. et al. Application of human factors to improve usability of clinical decision support for diagnostic decision-making: a scenario-based simulation study. BMJ Qual Saf 2020; 29 (04) 329-340
• 34 Dauber-Decker KL, Feldstein D, Hess R. et al. Snowball group usability testing for rapid and iterative multisite tool development: method development study. JMIR Form Res 2025; 9: e55316
• 35 Jacobsohn GC, Leaf M, Liao F. et al. Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments. Healthcare (Amst) 2022; 10 (01) 100598
• 36 Pierce RP, Eskridge B, Ross B, Wright M, Selva T. Impact of a vendor-developed opioid clinical decision support intervention on adherence to prescribing guidelines, opioid prescribing, and rates of opioid-related encounters. Appl Clin Inform 2022; 13 (02) 419-430
• 37 Sommers SW, Tolle HJ, Trinkley KE. et al. Clinical decision support to increase emergency department naloxone coprescribing: implementation report. JMIR Med Inform 2024; 12: e58276
• 38 Simpson MJ, Ritger C, Hoppe JA. et al. Implementation strategies to address the determinants of adoption, implementation, and maintenance of a clinical decision support tool for emergency department buprenorphine initiation: a qualitative study. Implement Sci Commun 2023; 4 (01) 41
• 39 Powers EM, Shiffman RN, Melnick ER, Hickner A, Sharifi M. Efficacy and unintended consequences of hard-stop alerts in electronic health record systems: a systematic review. J Am Med Inform Assoc 2018; 25 (11) 1556-1566
• 40 Van de Velde S, Heselmans A, Delvaux N. et al. A systematic review of trials evaluating success factors of interventions with computerised clinical decision support. Implement Sci 2018; 13 (01) 114
• 41 Hum RS, Cato K, Sheehan B. et al. Developing clinical decision support within a commercial electronic health record system to improve antimicrobial prescribing in the neonatal ICU. Appl Clin Inform 2014; 5 (02) 368-387
• 42 Wu R, Foster E, Zhang Q, Eynatian T, Mishuris R, Cordella N. Iterative development of a clinical decision support tool to enhance naloxone coprescribing. Appl Clin Inform 2025; 16 (01) 215-222
• 43 Wally MK, Thompson ME, Odum S, Kazemi DM, Hsu JR, Seymour RB. PRIMUM Group. Opioid prescribing for chronic musculoskeletal conditions: trends over time and implementation of safe opioid-prescribing practices. Appl Clin Inform 2023; 14 (05) 961-972
• 44 McCoy AB, Russo EM, Johnson KB. et al. Clinician collaboration to improve clinical decision support: the Clickbusters initiative. J Am Med Inform Assoc 2022; 29 (06) 1050-1059