PDF icon Download PDF
Open Access
CC BY-NC-ND 4.0 · Appl Clin Inform 2023; 14(03): 566-574
DOI: 10.1055/s-0043-1769911
Research Article

Lessons Learned from a National Initiative Promoting Publicly Available Standards-Based Clinical Decision Support

Rina V. Dhopeshwarkar
1   Department of Health Sciences, NORC at the University of Chicago, Bethesda, Maryland, United States
,
Maysoun Freij
2   Department of Health Care Evaluation, NORC at the University of Chicago, Bethesda, Maryland, United States
,
Melissa Callaham
1   Department of Health Sciences, NORC at the University of Chicago, Bethesda, Maryland, United States
,
Priyanka J. Desai
1   Department of Health Sciences, NORC at the University of Chicago, Bethesda, Maryland, United States
,
Michael I. Harrison
3   Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, United States
,
James Swiger
3   Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, United States
,
Edwin A. Lomotan
3   Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, United States
,
Chris Dymek
3   Center for Evidence and Practice Improvement, Agency for Healthcare Research and Quality, Rockville, Maryland, United States
,
Prashila Dullabh
1   Department of Health Sciences, NORC at the University of Chicago, Bethesda, Maryland, United States
› Author Affiliations

Funding This work is based on research conducted by NORC at the University of Chicago under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. HHSP233201500023I).
› Further Information
Also available ateRef
 
 PDF Download Permissions and Reprints

Abstract

Background Clinical decision support (CDS), which provides tools to assist clinical decision-making, can improve adherence to evidence-based practices, prevent medical errors, and support high-quality and patient-centered care delivery. Publicly available CDS that uses standards to express clinical logic (i.e., standards-based CDS) has the potential to reduce duplicative efforts of translating the same clinical evidence into CDS across multiple health care institutions. Yet development of such CDS is relatively new and its potential only partially explored.

Objectives This study aimed to describe lessons learned from a national initiative promoting publicly available, standards-based CDS resources, discuss challenges, and report suggestions for improvement.

Methods Findings were drawn from an evaluation of the Agency for Healthcare Research and Quality Patient-Centered Outcomes Research CDS Initiative, which aimed to advance evidence into practice through standards-based and publicly available CDS. Methods included literature and program material reviews, key informant interviews, and a web-based survey about a public repository of CDS artifacts and tools for authoring standards-based CDS.

Results The evaluation identified important lessons for developing and implementing standards-based CDS through publicly available repositories such as CDS Connect. Trust is a critical factor in uptake and can be bolstered through transparent information on underlying evidence, collaboration with experts, and feedback loops between users and developers to support continuous improvement. Additionally, while adoption of standards among electronic health record developers will make it easier to implement standards-based CDS, lower-resourced health systems will need extra support to ensure successful implementation and use. Finally, although we found the resources developed by the Initiative to offer valuable prototypes for the field, health systems desire more information about patient-centered, clinical, and cost-related outcomes to help them justify the investment required to implement standards-based, publicly available CDS.

Conclusion While the standards and technology to publicly share standards-based CDS have increased, broad dissemination and implementation remain challenging.


 

Keywords

decision support systems - electronic health records - standards-based CDS - implementation - facilitators and barriers

Background and Significance

Clinical decision support (CDS) facilitates value-based care by increasing adherence to evidence-based practices and supporting patient-centered decision-making.[1] [2] CDS uses targeted clinical knowledge and patient health information, can be computerized or not, and can provide patient-specific clinical care recommendations or evidence-based guidance to clinicians or directly to patients.[3] [4] Promoting CDS use is, therefore, in the public interest. Scaling use can be challenging because health systems need both time and resources to develop or purchase, deploy, and maintain CDS.[5] [6] Sharing CDS resources can help avoid duplication in developing CDS[7] and improve use across health systems by lowering its cost,[8] thus improving health equity by helping make CDS available to lower-resourced health systems.

Most health systems purchase CDS from electronic health records (EHR) developers, which is often proprietary and may not be interoperable across different EHR systems.[9] Developing and implementing EHR-agnostic CDS requires use of standards and resources that either allow for CDS knowledge artifact integration and execution within EHR systems, or support CDS use as a service that exchanges patient data with the EHR.[10] [11] See [Table 1] for description.[12] [13]

Table 1

Clinical decision support standards

Standard type

Examples

Standard language for expressing medical knowledge in clinical decision support (CDS) logic

Clinical Quality Language (CQL), Arden Syntax

Standard medical terminologies

Systemized Nomenclature of Medicine – Clinical Terms (SNOMED CT), Logical Observation Identifiers Names and Codes (LOINC)

Interoperability standards to support data exchange between CDS applications and electronic health records (EHRs)

Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR)

Standard that enables an EHR, during certain workflows, to invoke CDS services executed outside the EHR using application programming interfaces (APIs). The CDS service may use knowledge artifacts written in CQL or other languages. Data from the EHR is shared with the service as FHIR resources

CDS Hooks

Only two publicly available repositories make EHR-agnostic CDS artifacts and resources freely available. One, OpenCDS, is a collaborative community that leverages open-source tooling and services to support modular development of standards-based CDS.[14] Another, CDS Connect, is a resource the Agency for Healthcare Research and Quality (AHRQ) funds as part of its Patient-Centered Outcomes Research (PCOR) CDS Initiative. CDS Connect makes standards-based CDS artifacts and metadata (e.g., implementation guides) freely available to the public.[15] [16] Numerous organizations have contributed artifacts to the CDS Connect Repository that are in varying degrees of implementation readiness. Artifacts span clinical domains that include cardiovascular disease, preventive care, chronic pain management, mental health, and drug–drug interactions.[17] One example of a standards-based CDS artifact in CDS Connect is “Factors to Consider in Managing Chronic Pain: A Pain Management Summary,” which provides information for clinicians to consider when managing a patient's pain.[18]

This paper presents lessons learned about designing, developing, implementing, and using publicly available standards-based CDS from evaluating such efforts through the PCOR CDS Initiative and its components ([Table 2]).

Table 2

Initiative component descriptions

Initiative component

Description

Patient-Centered Clinical Decision Support (CDS) Learning Network

(2016–2020)

The Patient-Centered CDS Learning Network convened workgroups that produced frameworks and other learning resources to advance evidence into practice through CDS that is patient-centered

CDS Connect

(2016–2023)

CDS Connect consists of a Repository for open-source CDS, a prototype Authoring Tool interface for creating standards-based CDS logic expressions, and other open-source tooling to support the development of standards-based CDS

Quantifying Efficiencies Gained through Shareable CDS

(2018–2019)

The Quantifying Efficiencies project aimed to examine whether adoption and implementation of shareable CDS artifacts from CDS Connect resulted in efficiencies relative to in-house development and implementation of proprietary CDS

CDS Demonstration Projects[43] [44] [45] [46] [47]

(2019–2022)

Five investigator-led teams developed, tested, and implemented standards-based PC CDS in various clinical settings. The resulting artifacts have been or will be made available to the public through the CDS Connect Repository

Drawing from these lessons, we identify broader considerations for improving standards-based CDS uptake.


 

Methods

The evaluation used multiple data collection methods, including: (1) key informant interviews (KIIs), (2) a web-based survey about CDS Connect, and (3) Technical Expert Panel (TEP) comment summaries. NORC at the University of Chicago Institutional Review Board and the U.S. Office of Management and Budget reviewed all study procedures.

Key Informant Interviews

We interviewed 40 informants: Initiative leaders (23) and Initiative contributors, participants, and consumers (17). Semistructured interview guides probed CDS development experience, Initiative participation, and perspectives on shareable, standards-based CDS. Since our study was part of the Initiative evaluation, informants' experience developing or implementing standards-based, publicly available CDS was generally tied to their Initiative involvement. Interview recordings were used to develop transcripts for analysis. We developed our initial codebook from the program materials and interview guide, which we refined using NVivo software to thematically code each transcript.[19]


 

Web-Based Survey

We developed the survey instrument (available from author on request) from our program document review and preliminary KII findings, which we cognitively tested with five informaticists. We conducted the survey from March 23 to June 7, 2021, to understand how users engaged with CDS Connect resources and user perceptions about the resources' value. From the 713 surveys distributed, we received 79 completes (11% response rate). We used R statistical software to produce descriptive statistics and cross-tabs of respondents' roles and resource use.[20] CDS Connect survey results have the potential for positive selection bias, since respondents were known to be already at least somewhat familiar with CDS Connect.


 

Technical Expert Panel

A 22-member TEP provided insight into the current CDS landscape and helped synthesize findings. Panelists were selected based on their expertise and willingness to participate in periodic meetings related to AHRQ's PCOR CDS Initiative ([Table 3]).

Table 3

Technical Expert Panel panelist representation

Panelist type

Number of representatives

Federal agencies

4

Academic medical centers

3

Patient advocacy organizations

2

Health systems clinical staff and providers

2

Health plans and value-based purchasers

2

Quality standards and measure developers

1

We held two TEP meetings (in March and July 2022), at which we presented the evaluation findings and discussed key themes and recommendations. We then reviewed the meeting transcripts to identify lessons and other guidance relevant to the evaluation.


 

Analysis and Synthesis

We used triangulation to consider findings on similar topics, qualitative findings to contextualize quantitative findings, and literature to add context for the findings.[21] Following independent data analysis for each method, four study team members compared findings related to similar themes across data sources.[22] We then cross-referenced these findings to identify lessons learned about AHRQ's Initiative and potential implications for the broader field. Team discussions further refined evaluation themes and their implications for the broader field. We present findings organized into four CDS lifecycle stages: (1) identifying and choosing evidence to inform CDS design, (2) developing CDS, (3) implementing CDS, and (4) measuring the impact of CDS use.[23]


 

 

Results

[Table 4] summarizes the findings from our evaluation of the PCOR CDS Initiative.

Table 4

Summary of findings on publicly available standards-based clinical decision support

Identifying and choosing evidence to inform clinical decision support (CDS) design

Users wanted more easy-to-understand information about the trustworthiness of the underlying evidence/guidelines of publicly available CDS artifacts

CDS developers faced challenges related to intellectual property (IP) restrictions around using evidence-based guidelines in free, publicly available CDS. Many used guidelines in the public domain that federal agencies published

Developing CDS

Users/potential users of publicly available CDS suggested they would have greater trust if the CDS was developed in collaboration with or validated by guidelines developers or other experts in the underlying evidence base

Contributors to CDS Connect wanted to receive more information about the CDS artifacts they contributed, including if/how the artifacts are used and any recommendations for improvement

Implementing CDS

Informants found that most EHRs did not fully support CDS standards such as CQL, and significant time and staff resources were needed for custom development, configuration, and maintenance

Developers and/or users of standards-based CDS Connect artifacts emphasized the importance of getting support from executive leadership, clinical departments, and health IT department leadership to ensure that necessary staff and resources were available to implement the CDS tool

Informants noted that equitable use of CDS Connect artifacts required supplying lower-resourced settings with education materials and support, including clear implementation guides

Measuring CDS impact

Informants emphasized the importance of demonstrating improved health outcomes for patients and a return on investment (ROI) for publicly available, standards-based CDS from the perspective of multiple stakeholders—including health system executives, clinician users, and health IT developers whose buy-in is critical to success

Identifying and Choosing Evidence to Inform Clinical Decision Support Design

In the early stages of CDS design, developers work with multiple stakeholders to prioritize evidence-based findings for dissemination via CDS, including developing criteria and use cases for the CDS.[23] We found that developers looking to develop publicly available, standards-based CDS need to communicate to potential users understandable and trustworthy information on the underlying evidence for CDS. We also found that developers looking to share CDS publicly face challenges with the intellectual property rights related to evidence-based guidelines.

Need for Understandable Information on Trustworthiness of Evidence Informing the Clinical Decision Support

We found from our survey and interviews that CDS Connect Repository users wanted easily understood information describing CDS artifacts. Users said they reviewed available metadata—artifact goal, target population, evidence-based references, human-readable logic, cautions, pilot findings, implementation guides—to determine CDS fitness for use. Many found the metadata helped improve transparency and allowed for assessment of the CDS purpose and implementation readiness, although some said not all metadata allowed users to evaluate CDS' trustworthiness. Users sought more information on the trustworthiness of evidence/guidelines underlying the artifacts, including identification of clinical guideline authors, clinical professional association endorsement, or studies demonstrating clinical effectiveness. Users also noted limited information regarding the source of the underlying evidence base or guidelines, including when evidence was last updated. These findings demonstrate that the ability to review metadata, or data describing CDS artifacts, is critical for user trust and potential uptake.


 

Challenges of Intellectual Property Restrictions

Several interviewees reported that intellectual property licensing requirements for some evidence-based guidelines make it challenging for developers to use them for publicly available CDS. Many CDS development projects in the Initiative used guidelines published by the Centers for Disease Control and Prevention (CDC) or the U.S. Preventive Services Task Force, which alleviated some, but not all, concerns. Informants noted that early engagement with organizations regarding intellectual property restrictions is imperative, and whenever possible, public domain, creative commons, or open-source licensing agreements should be utilized. Informants also suggested guideline developers eliminate or streamline licensing restrictions to facilitate guideline adoption for open CDS artifacts.[24]


 

 

Developing Clinical Decision Support Artifacts

After evidence or clinical guidelines have been identified for CDS, developers must translate them into computable clinical knowledge,[23] which requires identifying key guideline elements often not specified to computable precision.[25] The development phase also includes testing and iterative refinement of CDS, to ensure it performs as expected and in a consistent manner.[4] When developing publicly available standards-based CDS, we found trust, once again, to be key, as well as the need for feedback loops between users and developers of such tools.

Trusting Knowledge Translation

In the survey, 37 out of 42 CDS Connect Repository users agreed that the Repository was valuable for making evidence-based CDS public and advancing the development of standards-based CDS. Yet, we found that to trust available artifacts, potential users need to trust the knowledge translation process. Part of this process is selecting the most appropriate codes and value sets to accurately capture criteria for which patients, conditions, therapies, and recommended clinical actions to include or exclude. Informants suggested that authors of standards-based CDS work in collaboration with guideline developers, standards development organizations (e.g., Regenstrief for Logical Observation Identifiers Names and Codes), and clinical experts, to appropriately address gaps in existing value sets and codes[26] and validate translated CDS logic expressions to ensure accuracy and increase trust in the translation process.


 

Need for Feedback Mechanisms

Several authors contributing to CDS Connect noted the importance of having a feedback mechanism between users and developers to help them understand if and how the artifacts are used and what improvements or updates could be made. This mechanism would also provide an opportunity to assess artifact quality and trustworthiness through a standard reporting process.


 

 

Implementing Clinical Decision Support

Once CDS tools are developed, they must be implemented into local systems for use. Though repositories may make standards-based CDS publicly available, the tools still require adaptation for use in local and often proprietary EHR environments. Implementing CDS interventions also promotes research and knowledge-sharing on the effective use of CDS for patients, caregivers, and other stakeholders.[23] For CDS Connect in particular, we found limited evidence that users were implementing available CDS. Only 7 of the 42 respondents who used the CDS Connect Repository indicated they had downloaded and adapted CDS artifacts with the intent of implementation. Among implementation challenges were technical difficulties implementing them in EHR systems and getting buy-in from leadership. In addition, we found making resources on implementation available to users promotes more equitable use of available CDS, particularly in low-resourced settings such as community health centers.

Difficulty Implementing in Electronic Health Record Systems

Sites attempting to implement CDS Connect artifacts invested significant time and staff resources to work with EHR developers to implement and maintain standards-based CDS. Most EHR developers do not support import or execution of Clinical Quality Language (CQL)-based knowledge artifacts.[27] Even among EHR developers that support CDS Hooks use for enabling CQL-based CDS as a service, CDS Hooks limits where CDS trigger points can be placed within the provider workflow. Additionally, EHR developers do not consistently offer FHIR services to support standards-based CDS implementation. Informants noted that two different EHR systems can use the same FHIR service but return different EHR data elements.

Therefore, implementing standards-based artifacts requires mapping standardized data elements to proprietary codes EHR developers use. The extent of mapping, which can only be determined with EHR developer input, relates to the specific EHR data elements the CDS tools must access, and whether those EHR data are accessible using FHIR resources. Due to the high degree of CDS customization and configuration requiring EHR developer support, implementers noted that health systems using standards-based CDS artifacts need to account for the additional resources required to maintain and update the CDS within specific EHR platforms.


 

Executive Leadership Support Critical to Implementation Success

CDS Connect artifact users noted that, before developing or implementing CDS within a health care institution, it is important to ensure buy-in from executive leaders who manage the staff and IT resources needed for adaptation, integration into EHR systems, and implementation. Users recommended selecting CDS that aligns with health system priorities. Investigators for AHRQ's Quantifying Efficiencies project found that “shared artifacts that explicitly state how they are designed to meet specific strategic needs (e.g., clinical priorities, regulatory mandates) may be more likely to be selected and implemented,” and tie-in with institutional priorities helped with receiving approvals from multiple committees.[28] To help obtain buy-in at all levels, users suggested that publicly available CDS metadata include enough information to allow health care executives and clinical leaders to assess alignment with organizational priorities.


 

Implementation Resources Necessary for Equitable Use

Publicly available, standards-based CDS hold potential for supporting more equitable distribution of CDS resources by making evidence-based CDS freely available. Accompanying metadata and resources (such as implementation guides) facilitate adaptation of available CDS for local environments, build trust in its evidence-base, and maximize the potential that the tools will be used effectively.[29] However, as noted, implementing such CDS still requires substantial investment of time and staff. More technical assistance and support are needed to enable lower-resourced health systems to adapt standards-based CDS in local environments.


 

 

Measuring Clinical Decision Support Impact

Measuring health care outcomes based on CDS dissemination and use is among the most important steps of the CDS lifecycle.[23] Measuring effectiveness is important to sustain, improve, and spread CDS use, as well as prohibit CDS-induced harm.[30] The few studies on the efficacy of publicly available, standards-based CDS generally focus on new products and the design or pilot stages of CDS development, rather than outcomes.[13] Yet, we found that demonstrating a return on investment (ROI) is critical to promoting CDS uptake.

Demonstrating Return on Investment Critical for Uptake

Users and potential users of CDS artifacts desired information regarding the impact of CDS Connect Repository artifacts on health outcomes. Informants noted that this information would build trust and encourage broader uptake of such CDS, including among health systems. CDS Connect includes information on the experience of implementing a few artifacts, including detailed pilot testing reports,[31] and anecdotal reports that “sites have implemented this CDS with no reported issues.” But these reports do not include measures that could help health systems determine whether CDS use improved clinical processes, workflow, or patient outcomes.

Initiative stakeholders emphasized the importance of demonstrating an overall ROI. This requires measuring and calculating the cost and resources required to develop, implement, and maintain CDS, and the benefits and improvements resulting from its use. Informants noted that ROI information must include multiple perspectives on success. For a patient, success may be facilitation of an informative health care discussion; for a clinician, success may be improved workflow efficiency; for a health system executive, success may be population-level improvements in quality outcomes. All these types of ROI outcomes go beyond those primarily reported by CDS researchers in the Initiative.


 

 

 

Discussion

Users agreed that resources made available through the Initiative are valuable for advancing the development of publicly available, standards-based CDS. However, several factors may hinder uptake of these resources and related CDS. Below, we consider other research in the field alongside our findings, to identify broader implications and insights for promoting the advancement and adoption of publicly available, standards-based CDS in health care.

Public Clinical Decision Support Repositories Should Include Clear Information on the Underlying Evidence Base

Trust in their integrity and validity is important when deciding to use CDS tools.[32] Platforms that provide publicly available CDS can bolster trust by providing accessible information that helps potential users understand and assess the evidence underlying the tools and their fitness for use. Proposed frameworks for trust in CDS include a rating system based on evidence assessments.[29] [33]


 

Guidelines in the Public Domain Are Good Candidates for Development into Publicly Available Clinical Decision Support

Navigating intellectual property or licensing restrictions around evidence-based guidelines can be an arduous process. Selecting guidelines in the public domain, such as those developed or published by federal agencies, can help avoid this.


 

A Collaborative Approach to Authoring Improves Trustworthiness of Publicly Available, Standards-Based Clinical Decision Support

Even if evidence-based sources and guidelines are trusted, their translation into CDS logic may not be—making validation of the translation process critical to avoid misinterpretation. Other studies have noted that adopters of standards-based CDS take steps to review CDS logic expressions prior to implementing them independently.[1] [28] A collaborative development process that includes guideline developers, standards development experts, patients,[34] and CDS developers can reassure patient and clinician users that CDS artifacts are vetted. CDC's Adapting Clinical Guidelines for the Digital Age Initiative and AHRQ's CDS Connect project have examples of a collaborative translation process that includes input from guideline developers and clinical experts.[35] An additional potential benefit of a collaborative approach is that stewards of CDS artifacts can work with guideline developers to ensure CDS artifacts are maintained to align with clinical guideline updates. The CDC Initiative also informed creation of the FHIR Clinical Guidelines Implementation Guide to support guideline developers' and CDS developers' joint efforts to develop computable representation of narrative clinical guidelines using FHIR.[36]


 

Feedback Loops Are Critical to Developing, Maintaining, Improving, and Trusting Publicly Available, Standards-Based Clinical Decision Support Artifacts

One benefit of publicly available open-source resources for CDS, including CDS artifacts in a repository such as CDS Connect, is that they can be iteratively improved and refined with feedback from a community of users. This has been demonstrated through the work of OpenCDS, which boasts over 500 members.[37] Publicly available repositories of standards-based CDS should incorporate mechanisms by which contributing CDS authors can receive input from other CDS authors and users. This input could inform iterative improvements for release in future artifact versions, which could protect against inadvertent dissemination of unusable or harmful tools.


 

Broader Clinical Quality Language Adoption by Electronic Health Record Developers May Ease Many Challenges Health Systems Currently Face when Implementing Standards-Based Clinical Decision Support

Because CQL is the standard Centers for Medicare and Medicaid Services currently requires for electronic Clinical Quality Measure specifications, EHR developers have widely supported CQL in the quality measurement domain. Even so, most EHR vendors have not adopted CQL as a CDS standard and still use proprietary expression languages.[27] EHR developers may not feel compelled to prioritize native support of CQL until knowledge artifact developers create a sufficient critical mass of CQL content. Regulation from the U.S. Office of the National Coordinator for Health Information Technology (ONC) requires adoption of FHIR Release 4 within EHR platforms as a condition of their certification under the Health IT Certification Program.[38] CQL is currently the only CDS standard supporting use of the FHIR as a data model, which may spur its adoption for CDS.[39] Wider CQL adoption by EHR developers would reduce the effort required to implement and maintain standards-based CDS, making it more accessible to lower-resourced settings. Increasing the availability of CQL-based knowledge artifacts such as those available in CDS Connect may support a smoother transition to CQL-based CDS within EHRs in the future.


 

Successful Implementation and Use of Publicly Available, Standards-Based Clinical Decision Support Requires “Buy-in” at Multiple Levels within a Health System

Adoption of standards that support interoperability of health care data brought with it the promise of applications that could be “plugged into” EHR systems and then swapped out for newer, more updated applications as needed. Users may expect standards-based CDS to provide the same agility to install and update CDS tools in EHR systems. However, the complex process of implementing standards-based CDS requires support from executive leadership, health IT departments, and clinicians—making buy-in at multiple levels within a health care system necessary for success in implementing standards-based CDS. The effort required for full stakeholder engagement may be the same as getting approval for developing and implementing custom-built CDS.


 

Lower-Resourced Settings Need Additional Support in Adapting and Implementing Publicly Available, Standards-Based Clinical Decision Support

Implementing standards-based CDS requires health IT staff familiar with CQL who can work with EHR developers to integrate and optimize the CDS tools, which often require multiple iterations and testing. The developer support required may result in additional implementation costs that put such CDS beyond the reach of lower-resourced health settings. Additional funding to support staff time, along with detailed implementation guides, may be necessary for publicly available, standards-based CDS to be equitably used across health care settings.


 

Further Studies Are Needed on the Short-Term (Process) Outcomes, Long-Term (Clinical) Outcomes, and Return on Investment of Adapting, Implementing, and using Publicly Available, Standards-Based Clinical Decision Support to Bolster Use

Providing this information can help build trust[2] [3] in CDS efficacy and help users decide whether CDS artifacts will help their health system improve care. However, the limited information available on outcome measures for publicly available CDS speaks to a broader issue. The paucity of CDS research studies that measure clinical outcomes may be because sustained CDS use and long-term patient follow-up are required to detect any significant clinical outcome impacts.[40]

ROI analyses could help justify investment in developing, implementing, using, and maintaining publicly available, standards-based CDS. However, ROI analyses of CDS systems and tools are rare and often rely on estimation—given the challenges of measuring direct, indirect, and intangible costs (e.g., burnout among clinician users) and recoverable gains from CDS implementation and use.[41] As noted, given that buy-in is often required at multiple levels of a health care system,[5] an ROI analysis must also consider multiple perspectives in determining the appropriate inputs to such calculations.


 

 

Conclusion

Our evaluation of AHRQ's PCOR CDS Initiative identified important lessons for developing and implementing publicly available standards-based CDS. We found that trust, a critical factor in uptake, can be bolstered through ensuring transparent information on underlying evidence, collaboration with experts in the development process, and feedback loops between users and developers to support continuous improvement. Additionally, while adoption of CDS standards among EHR developers will make it generally easier to implement standards-based CDS, lower-resourced health systems will need extra support to ensure successful implementation and use. Finally, health systems desire more information about outcomes and ROI—beyond assessments of the value and functionality of prototypes or their potential to improve health care decision-making or health outcomes—to help justify the cost and effort of implementing standards-based, publicly available CDS. Our findings closely align with recommendations prepared for the ONC Office of Clinical Quality Standards.[42] Their 2015 report highlighted barriers to CDS use, including high costs associated with implementing, customizing, and maintaining CDS. The report also called for feedback loops between CDS producers and users, more metadata to decrease uncertainty among clinician users of shared CDS resources, and efforts to demonstrate value. Future efforts to scale use of publicly available, standards-based CDS need to address persistent issues related to trust, implementation in commercial EHR systems, and CDS ROI measurement.


 

Clinical Relevance Statement

CDS has the potential to lower costs, improve efficiency, and reduce patient harm by providing timely information to clinicians, patients, and others.[2] Shareable CDS has the potential to reduce duplicative efforts of translating clinical evidence into CDS across health care institutions. This article describes the lessons learned about publicly available, standards-based CDS from AHRQ's PCOR CDS Initiative, which supported implementers, clinicians, and technology vendors developing shareable, patient-centered CDS tools.


 

Multiple-Choice Questions

  1. What may inhibit the use of a clinical guideline for publicly available CDS?

    • Intellectual property or licensing restrictions

    • The clinical guideline is published in the public domain by a federal organization such as the CDC

    • The clinical guideline is specified in a computable format

    • The clinical guideline aligns with health system priorities

    Correct Answer: The correct answer is option a.

  2. Why are feedback loops critical to publicly available, standards-based CDS?

    • They allow users to share real-world experiences with the artifact

    • They help authors update and improve the artifact

    • They can increase user trust

    • All of the above

    Correct Answer: The correct answer is option d.


 

 

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

The evaluation was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by the NORC at the University of Chicago Institutional Review Boards.


Authors' Contributions

All authors made substantial contributions to conception, design, and execution of this research. All authors participated in drafting the manuscript or revising it critically for important intellectual content and gave final approval of the version published.



Address for correspondence

Rina Dhopeshwarkar, MPH
NORC at the University of Chicago
4350 East-West Highway, 8th Floor, Bethesda, MD 20814
United States   

Publication History

Received: 22 December 2022

Accepted: 14 April 2023

Article published online:
26 July 2023

© 2023. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany


  Permissions and Reprints