Noninterruptive Clinical Decision Support Decreases Ordering of Respiratory Viral Panels during Influenza Season

Keywords

influenza - human - respiratory tract infections - multiplex polymerase chain reaction - decision support systems - clinical - cost control - computer-assisted decision-making - alert fatigue

Background and Significance

The utility of specific clinical testing for influenza is well established, in part due to the availability of rapid diagnostic testing as well as affordable and effective treatment such as oseltamivir and other neuraminidase inhibitors.[1] Rapid diagnosis of influenza in both outpatient and inpatient settings has led to a reduction in unnecessary antibiotics and additional tests such as chest X-rays and blood cultures.[2] [3] [4] Recently, the American Society for Clinical Pathology issued a Choosing Wisely recommendation discouraging routine ordering of broad respiratory pathogen panels that do not affect management, writing “consider first using tests of commonly suspected pathogens, which may change according to the location/season. Examples include rapid molecular or point-of-care tests for respiratory syncytial virus (RSV), influenza A/B, or Group A pharyngitis.”[5] Within the hospital setting, infection control teams often use respiratory viral panels (RVPs) to prevent nosocomial transmissions by quickly identifying and isolating infected patients. Some studies that examined the utility of RVPs for individual management of pediatric inpatients demonstrated a decreased duration of antibiotics,[6] [7] [8] shorter length of stay,[6] [7] and fewer chest X-rays.[8] However, Wishaupt et al found no difference in any of these outcomes when providers received rapid communication of positive results,[9] and McCullough et al found that while positive RVP results yielded decreased overall antibiotics, more patients were started on antibiotics when RVPs were performed.[3] Additionally, studies among adult hospitalizations and emergency department (ED) visits have found that RVP testing did not decrease antibiotics, length of stay, or overall cost.[4] [10] [11] [12] [13]

There are significantly fewer studies of RVP testing in the outpatient setting despite the fact that most encounters and antibiotic prescriptions for viral respiratory tract infections occur outside the hospital and ED. A study by Green et al that included outpatients found that while patients who tested positive for influenza virus received fewer antibiotics prescriptions, no effect was seen among patients testing positive for a noninfluenza virus.[14] This and other evidence supports the Choosing Wisely guideline for influenza testing over RVP testing in general populations with respiratory tract infection symptoms. Current studies do not address the utility of this testing in high-risk populations.

To actually improve adherence to guidelines and other best practices, health systems often use Clinical Decision Support Systems (CDSS), which have improved accuracy in cholesterol management,[15] medication-laboratory monitoring,[16] and warfarin dosing.[17] For reducing unnecessary laboratory testing specifically, Clinical Decision Support (CDS) techniques that intrinsically modify computerized physician order entry (CPOE)—such as interruptive alerts and cost display—have been shown to be superior to extrinsic techniques such as education, auditing, and feedback.[18] [19] Specifically, CPOE modifications have been shown to reduce duplicate inpatient laboratory orders,[20] improve adherence to guidelines for vitamin-D screening,[21] and reduce nonindicated testing such as erythrocyte sedimentation rate, urinalysis, glucose, international normalized ratio (INR), and others.[22] Additionally, while studies that reported economic outcomes were mixed, the results suggest that CDS is cost saving as well.[20] [23] [24] However, none of the aforementioned studies compared the effectiveness of interruptive versus noninterruptive methods. A review by Kawamoto and Lobach found that automatic, noninterruptive methods were more effective than interruptive alerts that require additional physician actions or data entry.[25] In fact, studies of interruptive alerts in drug safety monitoring found that they were overridden 49 to 96% of the time, suggesting that “alert fatigue” could also affect CDS systems targeting laboratory tests.[26] By contrast, simply changing CPOE search functions or naming conventions can reduce unintended laboratory tests such as 1,25-OH vitamin-D,[27] while a noninterruptive cost display can both reduce the overall number of tests and the associated cost.[28]

Objective

We aimed to decrease the proportion of RVPs to the total number of respiratory viral tests during influenza season. We used noninterruptive CDS methods including modifications of the CPOE search behavior and cost display of RVPs.

Methods

University of California, Los Angeles (UCLA) Health System is a large academic center that includes three licensed hospitals with two EDs and over 100 ambulatory offices including over 50 primary care clinics. In 2013, the institution implemented an Epic-based electronic health record (EHR) at all hospitals and ambulatory clinics (Epic Systems Corporation, Verona, Wisconsin, United States). There were two laboratory tests to evaluate for influenza: (1) the influenza polymerase chain reaction (PCR) test, which includes influenza A/B and (2) the RVP test, which is a multiplex respiratory viral PCR panel of 18 different tests, including influenza and RSV. Both tests were readily available in our EHR system's CPOE, without any additional decision support for our providers. When searching for a test with the keywords “virus” or “influenza,” the multiplex RVP would display in the same window as the triplex influenza test. Therefore, it was easy for providers to order an RVP regardless of the acuity level of their patients. Medicare reimbursement rates for Current Procedural Terminology (CPT) code 87633 (RVP) in 2019 was $571.72, and for 87631 (influenza PCR) was $175.98, for a cost difference of $395.74.

In November 2016, the two orders were combined into one reflex-based order titled “influenza AB RSV PCR reflex to resp virus panel, resp lower” and “influenza AB RSV PCR reflex to resp virus panel, resp upper.” These orders were given additional synonyms, including “flu” and “RSV” to encourage ordering of this less expensive reflex test; however, each individual test was still available in our system's CPOE for providers to order. Reflex testing is a common approach implemented to reduce the simultaneous ordering of both a screening test and the more definitive (and expensive) follow-up test (such as urinalysis and urine culture). The objective was to give clinicians the option to order the influenza PCR, followed by performance of the RVP if the influenza PCR was negative.

In November 2017, we eliminated reflex testing and modified the display names and search behavior of both tests. To discourage ordering of the RVP test, we added the text “immunocomp/critically ill patient only ($$$)” to the display name, thereby briefly reminding providers of recommended indications of this test. Additionally, we removed the search synonyms of the RVP test so that it would only appear in the search window by typing any of the terms in the order name “respiratory virus panel.” Conversely, to encourage appropriate ordering of the specific influenza and RSV PCR test, we added search synonyms including flu, flu virus, influenza virus, flu PCR, RSV virus, and RSV PCR. None of these changes involved any hard-stops, interruptive alerts, changes in ordering workflow, or changes to personalized preference lists for the individual user; they silently modified the availability and display of each test to encourage the more cost-effective of the two. [Table 1] displays the order names and synonym changes during the intervention. Costs to the institution due to the intervention were minimal, driven by the amount of time spent on planning and implementation for IT and laboratory analysts. Planning for this project occurred over a handful of brief meetings between informaticists, IT analysts, and laboratory staff, and the build/implementation phase required a month total of IT analyst time. Assuming full-time employment by all noncontractor team members, approximate cost is on the order of $5000 to $10,000.

Table 1 Respiratory viral panel and influenza test display name and search synonyms by intervention period

		March 2013–October 2016	November 2016–October 2017	November 2017–current
RVP test	Name	Respiratory virus panel	Influenza A and B and RSV PVR reflex to resp virus panel	Respiratory virus panel—immunocomp/critically ill patient ($$$)
	Synonyms	RVP	Flu, RSV, RVP	None
Influenza test	Name	Influenza A and B and RSV PVR	Influenza A and B and RSV PVR	Influenza A and B and RSV PVR
	Synonyms	Flu, RSV	Flu, RSV	Viral, Flu, influenza, flu PCR, RSV, and RSV PCR

Abbreviations: PCR, polymerase chain reaction; PVR, pulmonary vascular resistance; RSV, respiratory syncytial virus; RVP, respiratory viral panel.

Study Design

We conducted an interrupted time series design measuring the totals of both the RVP and influenza tests ordered within the UCLA Health System from March 2013 to April 2019. We included orders from all hospitals and EDs, as well as all ambulatory clinics. Therefore, the years 2013 to 2016 served as a baseline for comparison to the intervention years of 2017 to 2019. We used multivariable analysis to adjust for potential confounders and secular trends.

Outcome Measurement

The primary outcome measure is the proportion of RVPs to total respiratory viral testing (the sum of RVPs and influenza PCR tests) for each intervention period. Since there is no consensus on which patients should receive testing for respiratory viral symptoms, we focused only on those patients that received testing by examining which test was ordered. Recommended practice is to limit RVP orders to severely ill and/or immunocompromised patients, but even those parameters can be unclear and open to provider interpretation. Additionally, validation of the appropriateness of each order would be nearly impossible in a practical sense, given that we anticipate at least 50,000 orders during the study period. Therefore, examining the outcome as a proportion of costly viral testing to total viral testing eliminates the uncertainty of criteria for RVP testing.

Variable Definitions

[Table 2] defines all of the independent variables and covariates in this study. Time was included to control for any secular trends in the data. Season was included to control for any seasonal variation. Cardiorespiratory compromise and immunocompromised state were included specifically because of the verbiage included in the CDS intervention outlined in the innovation section. Since we were specifically reminding providers about these conditions, we felt it was necessary to include them in the model. Influenza-like illness or viral upper respiratory infection was included given the inclusion of influenza tests as part of the outcome. For all three diagnosis covariates, we used corresponding International Classification of Disease (ICD) codes for the encounter diagnosis for admissions and ED visits, and the order-associated diagnosis for office visits. Lastly, influenza incidence was gathered from the Los Angeles County Department of Public Health but has not been published in this form elsewhere (Acute Communicable Disease Control, County of Los Angeles Department of Public Health, 2019). However, this data include only the percentage of positive influenza tests from sentinel laboratories around Los Angeles County per month. We did not acquire total numbers of tests as that data were unavailable.

Table 2 Definitions of all independent variables and covariates. All variables were categorical and retrieved from University of California, Los Angeles's Clarity database with the exception of influenza incidence, which is continuous and retrieved from the Los Angeles County Department of Public Health

Independent variables	Details
Intervention period	Preintervention (2013–2016), reflex testing (2016–2017), and noninterruptive CDS (2017–2019)
Covariates	Details
Provider	Provider ID number
Time	Month to year of the ordering date
Season	Month of the ordering date
Clinical setting	Inpatient, ED, and ambulatory
Severe illness or cardiorespiratory compromise	Binary; at least 1 ICD-10 code among a list of various child concepts of parents R57 (shock), R65 (SIRS/sepsis), I46 (cardiac arrest), I50 (heart failure), J96 (respiratory failure), P27–29 (respiratory/cardiovascular conditions of the newborn)
Immunocompromised state	Binary; at least 1 ICD-10 code among: B20 (HIV), D80–84 (immunodeficiency), Z94 (transplant organ status)
Influenza-like illness or viral upper respiratory infection	Binary; at least 1 ICD-10 code among: J06.9 (acute upper respiratory infection), J9–11 (influenza), J20–22 (bronchiolitis/unspecified acute lower respiratory infection)
Influenza incidence	Percent of positive influenza PCR tests at sentinel laboratories that report to Los Angeles County

Abbreviations: CDS, clinical decision support; ED, emergency department; ICD, international Classification of Disease; PCR, polymerase chain reaction.

Analysis Plan

To evaluate the change in provider ordering of RVPs compared with influenza tests throughout each intervention period, we used a segmented regression analysis as described by Wagner et al.[29] Specifically, we used a mixed-effects logistic regression analysis on the dependent variable, the proportion of RVPs to the total number of respiratory viral tests. The random effects were represented by the provider covariate, which allowed for the evaluation of intraprovider behavior by clustering on ordering provider. Fixed effects included the intervention period, as well as the other covariates outlined in [Table 2].

Then, we performed the logistic regression model including all of the variables and covariates outlined in [Table 2], generating results as the odds ratio (OR) of ordering an RVP—including a 95% confidence interval—to convey the real-world effects of our intervention. We performed a second iteration of the full model with a releveled intervention period variable to compare the interventions to each other. Lastly, we performed a set of three validation models to evaluate for secular trends and intraperiod trends. Validation model 1 included only the intervention period, which evaluates for a change in levels representing the average proportion of RVP ordering in each intervention period. Validation model 2 added a continuous time variable, which evaluates for a general secular trend across the entire study period. Lastly, validation model 3 included an interaction term between intervention period and time, which evaluates for changes in trends, rather than levels as seen in validation model 1.

Results

From March 2013 to April 2019, there were 24,294 RVPs and 26,012 influenza tests for a grand total of 50,306 respiratory viral tests for the UCLA Health System ([Fig. 1]).

Fig. 1 Number of respiratory viral panels and influenza polymerase chain reaction tests per month from March 2013 to April 2019.

The baseline average proportion of RVPs to the total number of respiratory viral tests in the preintervention period was 61.6%. The proportion decreased to 44.7% in the reflex testing period and 32.6% in the noninterruptive CDS period ([Fig. 2]).

Fig. 2 Proportion of respiratory viral panels to total respiratory viral tests per month, with average proportions for each intervention.

The final logistic regression model included the previously mentioned variables, with ORs and 95% confidence intervals ([Table 3]). During the reflex testing period, the odds of ordering an RVP were 57% lower, while during the noninterruptive CDS period odds were 71% lower. When comparing the two interventions, odds were 33% lower during noninterruptive CDS. Additionally, the CDS period was not associated with an increase in overall numbers of RVPs, whereas reflex testing did result in such an increase as seen in [Fig. 1]. Validation model 2 found a weak secular time trend that was insignificant in magnitude compared with the change in average proportions in the full model.

Table 3 Mixed effects logistic regression final model of the proportion of respiratory viral panel orders out of total respiratory viral polymerase chain reaction orders

Final model variables	β	Standard Error	p-Value	OR	95% CI
Reflex testing[a]	−0.840	0.042	<0.0001	0.432	0.397–0.469
Noninterruptive CDS[a]	−1.235	0.059	<0.0001	0.291	0.259–0.327
CDS: reflex testing[b]	−0.395	0.037	<0.0001	0.673	0.668–0.679
Time	0.008	0.001	<0.0001	1.008	1.006–1.0110
Influenza incidence	−0.014	0.002	<0.0001	0.986	0.981–0.991
ED: inpatient setting	−1.195	0.068	<0.0001	0.303	0.265–0.346
Ambulatory: inpatient setting	−0.416	0.044	<0.0001	0.660	0.605–0.720
Severe illness	−0.017	0.046	0.715	0.983	0.899–1.076
Immunocompromised	0.244	0.038	<0.0001	1.276	1.185–1.374
Influenza-like illness	−0.351	0.057	<0.0001	0.704	0.629–0.788

Abbreviations: CDS, clinical decision support; CI, confidence interval; ED, emergency department; OR, odds ratio.

^a Primary predictor variables (intervention periods) compared with the preintervention period.

^b We used a releveled iteration of this model to compare the noninterruptive CDS intervention to the reflex testing intervention. The ED and ambulatory settings are both comparisons to the inpatient setting.

Clinical setting, patient factors, and influenza prevalence had statistically significant effects on RVP ordering. The odds of ordering an RVP were 34.0% less likely in the ambulatory setting, and 69.7% less likely in the ED, compared with the inpatient setting throughout the duration of the study. Increasing influenza incidence decreased odds of ordering an RVP by 1.4%, while the presence of an ICD-10 code for immunocompromised state increased odds by 27.6%.

Discussion

In this interrupted time series design, we noted a statistically significant decrease in the proportion of full RVP ordering out of total respiratory viral testing for both the reflex testing and noninterruptive CDS interventions. Of the two interventions, noninterruptive CDS was more effective in reducing the proportion of RVPs when compared with both the preintervention period and the reflex testing period. While there was a weak secular time trend across the duration of the study, the magnitude of this trend did not explain the significant change in the proportion of RVPs associated with the intervention periods.

Several covariates confered a statistically significant effect on the odds of RVP ordering, including the ambulatory and ED clinical settings, which decreased the odds of ordering an RVP significantly. A likely explanation is that, in deciding on immediate treatment interventions, the ED rarely needs to know the specific viral pathogen beyond influenza or RSV. As expected, the odds of ordering an RVP were 29.6% less with a diagnosis of an influenza-like illness or acute upper respiratory infection ICD-10 code. Unexpectedly, influenza incidence had a modest effect, with only a very slight decrease in the odds of RVP ordering as influenza incidence increases. We anticipated that influenza incidence would be a larger driver of the proportion of RVPs; however, since we only have monthly data, perhaps there was not enough variability. As mentioned previously, influenza incidence was measured as the percentage of positive influenza PCR tests reported from sentinel laboratories throughout Los Angeles County between the months of September and April. The influenza monitoring program does not collect data from May to August. The incidences of influenza in September and April were fairly consistent each year, so it is unlikely that any particular influenza season had an effect on ordering behavior. The only covariate with an increased likelihood of RVP ordering was the diagnosis of an immunocompromised state, which was expected given the verbiage included in the noninterruptive CDS naming convention. Lastly, the diagnosis of severe illness or cardiorespiratory compromise had no statistically significant effect on RVP odds despite the order name listing “critically ill” as an indication for RVP.

Odds of providers ordering an RVP were around 71% lower during the noninterruptive CDS intervention period. Although costs were not directly addressed in our study design, by comparing the actual RVP ordering rates to the projected rates as if no interventions had occurred, we can estimate the cost savings of each intervention to the UCLA Health System. Since Medicare reimbursement rates are a commonly used proxy for cost, we used the 2019 rates for CPT code 87633 (RVP: $571.72) and CPT 87631 (influenza PCR: $175.98), for a cost difference of $395.74. For the reflex testing period, the actual 44.7% rate compared with the projected 61.6% preintervention rate led to a projected 1,904 fewer RVPs ordered per year. Using the Medicare rates, this yields an estimated averted cost of $753,518 per year. For the noninterruptive CDS period, the actual 32.6% rate compared with the preintervention rate led to a projected 4,773 fewer RVPs per year, and thus an estimated averted cost of $1,259,474 per year. These direct cost estimates do not include additional indirect costs, such as the cost of additional interventions based on test results. However, even a conservative estimate of the effect of the interventions demonstrates significant potential cost savings.

The significant decrease in proportion of RVP ordering and its associated cost saving estimates demonstrate the power of noninterruptive CDS interventions. Interruptive CDS measures invite alert fatigue and provider burnout, whereas noninterruptive methods can clearly affect provider behavior with less potential for frustration. Additionally, spending limited information technology (IT) resources on labor-intensive interruptive alerts that have up to a 96% override rate may be wasteful. Alerts require careful planning in terms of trigger rules and target users, and often need regular content updates. While targeted educational campaigns can be effective in reducing unnecessary ordering, this also requires IT resources and risks burdening providers even further. By focusing on silent, seemless CDS interventions, we can nudge providers toward best practices without competing with other education campaigns for provider time and attention.

This study has limitations. The data come from a single EHR vendor at a single large health care system, which limits generalizability. Additionally, we did not collect information on the ordering provider, including level of training, department, etc., which may have revealed specific practice patterns that could be addressed with targeted education. We also did not examine patterns of antibiotic use as a result of these orders. The initial request for this project came from our institution's Department of Pathology and Laboratory Medicine, so we chose to focus on laboratory testing only. Examining the effect of our intervention on antibiotic use and performing a more complete cost analysis is a future study by our group. By using a retrospective time series rather than a randomized study, there is also a risk of unmeasured confounders. However, we attempted to include relevant covariates and controlled for secular trends in our model. Finally, our primary outcome measured changes in proportions of total respiratory viral PCR orders, as opposed to changes in proportions of clinically appropriate respiratory viral PCR orders, such as the number of patients who should have received an influenza test but instead received an RVP or no test at all. While we could use statistical software to identify the cohort with indications for an influenza test, reaching consensus on the criteria would be difficult given the lack of established protocols and significant clinical complexity.

Conclusion

In reducing unnecessary RVP ordering during influenza season, noninterruptive CDS was more effective than reflex testing. The noninterruptive CDS innovation described here included a modification of the laboratory tests' CPOE search synonyms and display names, a relatively minor change compared with the complex performance of an alert. This type of intervention could be applied to other laboratory tests that have high rates of inappropriate orders, and potentially avert more unnecessary cost to the health system.

Clinical Relevance Statement

This study demonstrates the impact of CDS on cost reduction and waste. In particular, it demonstrates the power of noninterruptive methods, which do not require complex logic and mitigates alert fatigue. Similar methods can be applied to other unnecessary testing practices at other institutions with certified EHR technology.

Multiple Choice Questions

1. What is a major barrier to clinician acceptance of CDS systems that utilize interruptive alerts?

   • Poor interoperability.

   • Alert fatigue.

   • Inability to link to standard guidelines.

   • Additional required data entry.

   Correct Answer: The correct answer is option b, alert fatigue. Alert fatigue is evident in that users override interruptive alerts up to 96% of the time. Standard examples include mild or insignificant drug–drug interaction alerts that are well known to users and considered at the time of prescription. Health care institutions should design alerts that identify high risk but low-frequency events that can facilitate interventions by the user. Poor interoperability is an issue for CDS systems; however, the CDS hooks specification can help standardize CDS modalities across institutions. This is an issue for design of CDS, but not for acceptance by end users. Inability to link to guidelines can be an issue for end users; however, it is not a common barrier to acceptance as most guidelines are readily available online or are seen as a burden to access. Additional required data entry is a barrier to clinician acceptance; however, it is not as significant an issue as alert fatigue.

2. Which of the following is an appropriate indication for a multiplex RVP?

   • A previously healthy adult presenting to an ambulatory clinic with fever and myalgias.

   • An asymptomatic child with chronic lung disease who has a household contact with diagnosed influenza.

   • Identifying the source of a respiratory viral illness spreading in an intensive care unit.

   • A previously healthy child seen in the ED for uncomplicated bronchiolitis.

   Correct Answer: The correct answer is option c. RVPs are clearly useful in identifying the viral agent responsible for respiratory viral outbreaks in a hospital, and can be used to cluster certain patients together to control the infection. Outside of infection control issues, RVPs do not have clear benefits in the management of simple respiratory illnesses. An RVP may be indicated in complex patients or those with severe illness that does not follow typical viral illness patterns. Option a is therefore incorrect because an influenza test would be more appropriate. Option b is incorrect because even though the child has an underlying illness, there are no current symptoms. Option d is incorrect because identifying the underlying viral etiology of bronchiolitis would not likely change management.

• References

• 1 Lee JJ, Verbakel JY, Goyder CR. , et al. The clinical utility of point-of-care tests for influenza in ambulatory care: a systematic review and meta-analysis. Clin Infect Dis 2019; 69 (01) 24-33
• 2 Bonner AB, Monroe KW, Talley LI, Klasner AE, Kimberlin DW. Impact of the rapid diagnosis of influenza on physician decision-making and patient management in the pediatric emergency department: results of a randomized, prospective, controlled trial. Pediatrics 2003; 112 (02) 363-367
• 3 McCulloh RJ, Andrea S, Reinert S, Chapin K. Potential utility of multiplex amplification respiratory viral panel testing in the management of acute respiratory infection in children: a retrospective analysis. J Pediatric Infect Dis Soc 2014; 3 (02) 146-153
• 4 Yee C, Suarthana E, Dendukuri N, Nicolau I, Semret M, Frenette C. Evaluating the impact of the multiplex respiratory virus panel polymerase chain reaction test on the clinical management of suspected respiratory viral infections in adult patients in a hospital setting. Am J Infect Control 2016; 44 (11) 1396-1398
• 5 Miller JM, Binnicker MJ, Campbell S. , et al. A guide to utilization of the microbiology laboratory for diagnosis of infectious diseases: 2018 update by the Infectious Diseases Society of America and the American Society for Microbiology. Clin Infect Dis 2018; 67 (06) e1-e94
• 6 Schulert GS, Lu Z, Wingo T, Tang YW, Saville BR, Hain PD. Role of a respiratory viral panel in the clinical management of pediatric inpatients. Pediatr Infect Dis J 2013; 32 (05) 467-472
• 7 Rogers BB, Shankar P, Jerris RC. , et al. Impact of a rapid respiratory panel test on patient outcomes. Arch Pathol Lab Med 2015; 139 (05) 636-641
• 8 Subramony A, Zachariah P, Krones A, Whittier S, Saiman L. Impact of multiplex polymerase chain reaction testing for respiratory pathogens on healthcare resource utilization for pediatric inpatients. J Pediatr 2016; 173: 196-201
• 9 Wishaupt JO, Russcher A, Smeets LC, Versteegh FG, Hartwig NG. Clinical impact of RT-PCR for pediatric acute respiratory infections: a controlled clinical trial. Pediatrics 2011; 128 (05) e1113-e1120
• 10 Oosterheert JJ, van Loon AM, Schuurman R. , et al. Impact of rapid detection of viral and atypical bacterial pathogens by real-time polymerase chain reaction for patients with lower respiratory tract infection. Clin Infect Dis 2005; 41 (10) 1438-1444
• 11 Hernes SS, Hagen E, Quarsten H, Bjorvatn B, Bakke PS. No impact of early real-time PCR screening for respiratory viruses on length of stay and use of antibiotics in elderly patients hospitalized with symptoms of a respiratory tract infection in a single center in Norway. Eur J Clin Microbiol Infect Dis 2014; 33 (03) 359-364
• 12 Mulpuru S, Aaron SD, Ronksley PE, Lawrence N, Forster AJ. Hospital resource utilization and patient outcomes associated with respiratory viral testing in hospitalized patients. Emerg Infect Dis 2015; 21 (08) 1366-1371
• 13 Semret M, Schiller I, Jardin BA. , et al. Multiplex respiratory virus testing for antimicrobial stewardship: a prospective assessment of antimicrobial use and clinical outcomes among hospitalized adults. J Infect Dis 2017; 216 (08) 936-944
• 14 Green DA, Hitoaliaj L, Kotansky B, Campbell SM, Peaper DR. Clinical utility of on-demand multiplex respiratory pathogen testing among adult outpatients. J Clin Microbiol 2016; 54 (12) 2950-2955
• 15 Scheitel MR, Kessler ME, Shellum JL. , et al. Effect of a novel clinical decision support tool on the efficiency and accuracy of treatment recommendations for cholesterol management. Appl Clin Inform 2017; 8 (01) 124-136
• 16 Lau B, Overby CL, Wirtz HS, Devine EB. The association between use of a clinical decision support tool and adherence to monitoring for medication-laboratory guidelines in the ambulatory setting. Appl Clin Inform 2013; 4 (04) 476-498
• 17 Melton BL, Zillich AJ, Saleem J, Russ AL, Tisdale JE, Overholser BR. Iterative development and evaluation of a pharmacogenomic-guided clinical decision support system for warfarin dosing. Appl Clin Inform 2016; 7 (04) 1088-1106
• 18 Delvaux N, Van Thienen K, Heselmans A, de Velde SV, Ramaekers D, Aertgeerts B. The effects of computerized clinical decision support systems on laboratory test ordering: a systematic review. Arch Pathol Lab Med 2017; 141 (04) 585-595
• 19 Kobewka DM, Ronksley PE, McKay JA, Forster AJ, van Walraven C. Influence of educational, audit and feedback, system based, and incentive and penalty interventions to reduce laboratory test utilization: a systematic review. Clin Chem Lab Med 2015; 53 (02) 157-183
• 20 Bates DW, Kuperman GJ, Rittenberg E. , et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med 1999; 106 (02) 144-150
• 21 Felcher AH, Gold R, Mosen DM, Stoneburner AB. Decrease in unnecessary vitamin D testing using clinical decision support tools: making it harder to do the wrong thing. J Am Med Inform Assoc 2017; 24 (04) 776-780
• 22 Krasowski MD, Chudzik D, Dolezal A. , et al. Promoting improved utilization of laboratory testing through changes in an electronic medical record: experience at an academic medical center. BMC Med Inform Decis Mak 2015; 15: 11
• 23 Smith DH, Feldstein AC, Perrin NA. , et al. Improving laboratory monitoring of medications: an economic analysis alongside a clinical trial. Am J Manag Care 2009; 15 (05) 281-289
• 24 Khan S, Maclean CD, Littenberg B. The effect of the Vermont Diabetes Information System on inpatient and emergency room use: results from a randomized trial. Health Outcomes Res Med 2010; 1 (01) e61-e66
• 25 Kawamoto K, Lobach DF. Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA Annu Symp Proc 2003; 2003: 361-365 ; Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1480005/pdf/amia2003_0361.pdf . Accessed October 12, 2018
• 26 van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (02) 138-147
• 27 White AA, McKinney CM, Hoffman NG, Sutton PR. Optimizing vitamin D naming conventions in computerized order entry to support high-value care. J Am Med Inform Assoc 2017; 24 (01) 172-175
• 28 Goetz C, Rotman SR, Hartoularos G, Bishop TF. The effect of charge display on cost of care and physician practice behaviors: a systematic review. J Gen Intern Med 2015; 30 (06) 835-842
• 29 Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther 2002; 27 (04) 299-309