Appl Clin Inform 2018; 09(01): 163-173
DOI: 10.1055/s-0038-1632397
State of the Art/Best Practice Paper
Schattauer GmbH Stuttgart

The Value of Monitoring Clinical Decision Support Interventions

Eileen Yoshida
Shirley Fei
Karen Bavuso
Charles Lagor
Saverio Maviglia
Further Information

Publication History

31 October 2017

08 January 2018

Publication Date:
07 March 2018 (online)


Background Well-functioning clinical decision support (CDS) can facilitate provider workflow, improve patient care, promote better outcomes, and reduce costs. However, poorly functioning CDS may lead to alert fatigue, cause providers to ignore important CDS interventions, and increase provider dissatisfaction.

Objective The purpose of this article is to describe one institution's experience in implementing a program to create and maintain properly functioning CDS by systematically monitoring CDS firing rates and patterns.

Methods Four types of CDS monitoring activities were implemented as part of the CDS lifecycle. One type of monitoring occurs prior to releasing active CDS, while the other types occur at different points after CDS activation.

Results Two hundred and forty-eight CDS interventions were monitored over a 2-year period. The rate of detecting a malfunction or significant opportunity for improvement was 37% during preactivation and 18% during immediate postactivation monitoring. Monitoring also informed the process of responding to user feedback about alerts. Finally, an automated alert detection tool identified 128 instances of alert pattern change over the same period. A subset of cases was evaluated by knowledge engineers to identify true and false positives, the results of which were used to optimize the tool's pattern detection algorithms.

Conclusion CDS monitoring can identify malfunctions and/or significant improvement opportunities even after careful design and robust testing. CDS monitoring provides information when responding to user feedback. Ongoing, continuous, and automated monitoring can detect malfunctions in real time, before users report problems. Therefore, CDS monitoring should be part of any systematic program of implementing and maintaining CDS.


All the authors were affiliated with Clinical Informatics, Partners HealthCare System Inc., Boston, Massachusetts, United States at the time of the study.

Protection of Human and Animal Subjects

Neither humans nor animal subjects were included in this project.