Clinician Perceptions of a Computerized Decision Support System for Pediatric Type 2 Diabetes Screening

Abstract

Objective With the increasing prevalence of type 2 diabetes (T2D) in youth, primary care providers must identify patients at high risk and implement evidence-based screening promptly. Clinical decision support systems (CDSSs) provide clinicians with personalized reminders according to best evidence. One example is the Child Health Improvement through Computer Automation (CHICA) system, which, as we have previously shown, significantly improves screening for T2D. Given that the long-term success of any CDSS depends on its acceptability and its users' perceptions, we examined what clinicians think of the CHICA diabetes module.

Methods CHICA users completed an annual quality improvement and satisfaction questionnaire. Between May and August of 2015 and 2016, the survey included two statements related to the T2D-module: (1) “CHICA improves my ability to identify patients who might benefit from screening for T2D” and (2) “CHICA makes it easier to get the lab tests necessary to identify patients who have diabetes or prediabetes.” Answers were scored using a 5-point Likert scale and were later converted to a 2-point scale: agree and disagree. The Pearson chi-square test was used to assess the relationship between responses and the respondents. Answers per cohort were compared using the Mann–Whitney U-test.

Results The majority of respondents (N = 60) agreed that CHICA improved their ability to identify patients who might benefit from screening but disagreed as to whether it helped them get the necessary laboratories. Scores were comparable across both years.

Conclusion CHICA was endorsed as being effective for T2D screening. Research is needed to improve satisfaction for getting laboratories with CHICA.

Keywords

screening - obesity - diabetes mellitus - clinical decision support - testing - evaluation

Background and Significance

The prevalence of type 2 diabetes (T2D) is increasing in youth. Between 2001 and 2009, there was a 30% rise in the prevalence of pediatric T2D in the United States, and it is estimated that there will be a fourfold increase in prevalence by 2050.[1] Moreover, approximately 15% of adolescents have severe (class 2 or 3) obesity, putting them at an increased risk of T2D.[2] The American Diabetes Association (ADA) recommends screening youth who meet high-risk criteria to enable an early detection and timely treatment of T2D.[3] [4] [5] The ADA also recommends instituting primary prevention efforts, specifically lifestyle modification with a clinical follow-up for youth with obesity and glucose levels that are elevated but not diagnostic of diabetes (i.e., prediabetes).[5] However, screening and intervention practices are limited due to primary care clinicians' lack of knowledge of the guidelines and the absence of systems that support completion of testing and follow-up appointments.[6] [7]

We implemented published screening guidelines for T2D in pediatric primary care practices by using a clinical decision support system (CDSS): Child Health Improvement through Computer Automation (CHICA) system.[8] The CHICA diabetes-screening module was implemented in a cluster randomized clinical trial (RCT) performed in four pediatric clinics that included 1,369 patients. Computerized clinical decision support significantly increased the rates of screening compared with control clinics (31.4 vs. 9.2%; adjusted odds ratio [OR]: 4.6; 95% confidence interval [CI]: 1.5–14.7) and was associated with a greater proportion of youth attending a scheduled follow-up appointment (29.4 vs. 18.9%; adjusted OR: 1.8; 95% CI: 1.5–2.2).[8]

Objectives

The ongoing success of a CDSS depends on its acceptability by and usefulness to clinical care providers, health care staff, and patients. Nevertheless, acceptability results are infrequently reported among computerized decision support users despite the need to understand user and system needs for improvement. As part of routine quality improvement of CHICA, we annually administer a survey to CHICA users. In this study, we examined responses to items specific to the usability and perceived efficacy of the CHICA diabetes-screening module for identifying and obtaining laboratory tests for youth who meet published criteria for screening for T2D.

Methods

Child Health Improvement through Computer Automation

CHICA and its use in pediatric clinics have been previously described.[9] [10] [11] Briefly, CHICA is a CDSS that works as an add-on to the electronic medical record (EMR). CHICA uses standard HL7 (health level 7) messaging, web services, and standard Arden medical logic modules (MLMs), leveraging patient data to generate tailored patient screening forms (prescreener forms, available in English and Spanish) that caregivers and/or patients complete in the waiting room using tablets ([Supplementary Appendix A], available in the online version). These forms are uploaded in real time to the CHICA server, which combines answers with medical history from the medical record as well as previous prescreener form answers. CHICA's MLMs have embedded priority scores to produce a prioritized list of recommendations for health care providers (physicians and advance practice providers referred to as “providers” for the purposes of CHICA). These scores are based on: (1) the probability that the patient has a condition, (2) the seriousness of this condition, (3) the effectiveness of alerting the physician to this condition, and (4) the evidence of the intervention's effectiveness.[12] The top six prioritized recommendations are displayed, along with vital signs, growth data, and an area to document the physical examination on the Provider Worksheet displayed as an interactive window within the EMR ([Supplementary Appendix B], available in the online version). For each of the prioritized recommendations, providers can document their response by checking boxes corresponding to what actions were taken. Nonhealth care provider clinic personnel (referred to as “staff” for the purposes of CHICA) also interact with CHICA by distributing electronic tablets to patients to fill out the prescreener forms, as well as any informational handouts that are generated by CHICA to reinforce provider counseling and provide educational information.

The CHICA diabetes-screening module included the following pertinent questions on the prescreener form that the caregivers answered: “Did [patient's name] mother ever have a baby born over 9 pounds?”; “Did [patient's name] mother ever have diabetes during pregnancy?”; “Does [patient's name] have a parent or grandparent with diabetes?” Answers to these questions were combined with data on sex, race, height, and weight that CHICA collected from the EMR to determine if each patient met the ADA published criteria for screening for T2D: age of 10 years and older with a body mass index of ≥85th percentile and two more risk factors for T2D.[13] If these data indicated that the patient was at an increased risk of T2D, CHICA generated a prompt to the physician to assess for diabetes symptoms and signs of insulin resistance, while also recommending screening and follow-up steps.

Surveys

Beginning in 2011, the CHICA administrators implemented an annual quality improvement and satisfaction survey that consisted of 12 core questions on user acceptability.[14] During the RCT of the CHICA diabetes-screening module, two additional questions were added to the annual quality improvement and satisfaction survey. CHICA users (providers and staff) were asked to rate two T2D screening-related statements using a pencil/paper format: (1) “CHICA improves my ability to identify patients who might benefit from screening for T2D” (the “screening question”) and (2) “CHICA makes it easier to get the lab tests necessary to identify patients who have diabetes or prediabetes” (the “labs question”). Since these diabetes-related questions were for 2015 and 2016, the collected data were from users who had experience with CHICA for at least a year between May 2014 and 2015 and between May 2015 and 2016, respectively.

Participants and Data Collection

Between 2013 and 2016, we conducted an RCT to assess the effectiveness of the diabetes-screening module within CHICA. For this study, we randomized two primary care pediatric clinics to receive CHICA plus the diabetes-screening module and two other primary care pediatric clinics to receive CHICA without the diabetes-screening module. The latter clinics served as control clinics for the RCT, and hence their providers and staff did not answer the survey questions analyzed for the present report.[8] Only providers and staff from the two clinics that had access to the CHICA diabetes-screening module in the RCT were asked to respond to the T2D module satisfaction questions. Answers were provided on a 5-point Likert scale as follows: (1) strongly disagree, (2) somewhat disagree, (3) neutral, (4) somewhat agree, and (5) strongly agree. To encourage candid responses, we did not collect identifiable information (e.g., name, date of birth, employee ID). We collected data from both years of 2015 and 2016 with the intent to compare data from years since acceptability changes over time.[14] Trained research assistants entered the data into REDCap (Vanderbilt University, Nashville, Tennessee, United States), a web-based survey tool.[15]

Data Analysis

To assess the relationship between respondents and their perceptions, we performed a chi-square test of independence. For the purposes of this analysis, we converted the 5-point Likert system into a 2-point system, with (1) strongly disagree, (2) somewhat disagree, and (3) neutral coded as disagree, and (5) strongly agree and (4) somewhat agree coded as agree. We opted for the scale conversion for two reasons: we wanted to look at the responses as binary (perceives benefit/does not perceive benefit) and we wanted to be sure that all of the cell counts during analysis would be above 5 to meet the assumptions of Pearson's chi-square test. We included “neutral” in the disagree bin to achieve conservative estimates in our results. We wanted to compare answers across the 2 years of the study because our previous work has shown that CHICA users' perceptions of CHICA improve with experience.[14] To compare the answers across both years, we used the crude scores of the 5-point Likert scale. Because the scores are ordinal and because the data were nonparametric, we used the Mann–Whitney U test. We used SPSS (Statistical Package for the Social Science, Version 26, IBM Corp., Armonk, New York, United States) to conduct the statistical analyses. The Indiana University Institutional Review Board approved this study.

Results

A total of 60 respondents completed the annual quality improvement and satisfaction survey over both years (2015 and 2016). The characteristics of the respondents are shown in [Table 1]. The response rate was 100% for both years (research assistants followed up on every survey distributed). The majority of the respondents were providers: physicians and nurse practitioners. Physician providers comprised pediatricians (51.6%), combined medicine–pediatrics (16.1%), and family medicine physicians (32.3%).

Table 1 Respondents characteristics

Type Of Provider	2015 N (%), N = 32	2016 N (%), N = 28
Physicians/NPs	16 (50.0)	16 (57.1)
Nurse	5 (15.6)	3 (10.7)
Medical assistant	11 (34.4)	7 (25.0)
Other a	0 (0.0)	2 (7.1)
Physicians' specialties
Pediatrics	8 (50.0)	8 (53.4)
Medicine–Pediatrics	2 (12.5)	3 (20.0)
Other	6 (37.5)	4 (26.6)
Time at the clinic
Full time	19 (59.4)	18 (64.3)
Part time	13 (40.6)	10 (35.7)

Abbreviation: NP, nurse practitioner.

^a Other includes clinical administrators, front desk personnel, and care coordinators.

[Fig. 1] presents the percentage of respondents who agreed or did not agree with the statements: “CHICA improves my ability to identify patients who might benefit from screening for T2D” and to “CHICA makes it easier to get the lab tests necessary to identify patients who have diabetes or prediabetes.” The majority of providers and clinic staff agreed that CHICA improved their ability in identifying patients at risk of T2D, 71% for providers and 57% for clinic staff ([Fig. 1], left panel), but did not agree that it improved their ability to obtain laboratories, 58% of providers and 55% of clinic staff ([Fig. 1], right panel). Providers and staff did not differ significantly in their answers regarding either the screening question response (p = 0.20) or the laboratory question response (p = 0.52).

Fig. 1 Left: responses of clinicians to the screening question. Right: responses of the clinicians to the laboratories question.

The crude questionnaire scores from both years are shown in [Fig. 2]. Responses to the screening question (p = 0.31) and the laboratories question (p = 0.59) did not change between the years 2015 and 2016.

Fig. 2 The results of independent samples Mann–Whitney U test to the screening (p = 0.31) and laboratory (p = 0.59) questions over the years of 2015 and 2016.

Discussion

This study presents repeated data on the perceived efficacy of the CHICA diabetes-screening CDSS that was used as part of a randomized controlled clinical trial. In that trial, the CHICA diabetes screening CDSS was associated with a quadrupling of the odds of screening for prediabetes and T2D in youth meeting published criteria compared with control clinics, and was associated with a greater proportion of youth attending a scheduled follow-up appointment.[8] It follows that providers and staff members agreed that having the T2D module facilitated screening appropriate youth. However, providers and staff did not agree that the CHICA diabetes screening CDSS facilitated getting recommended laboratory tests.

Belamarich et al discussed how pediatricians have too many clinical recommendations to complete in a limited clinical encounter.[16] It is challenging to keep up with the numerous recommendations within the allotted 15 to 20 minutes per patient, especially when providers are not aware of screening guidelines. Even when they are aware of the need for screening, they are not likely to be on the list of concerns of the caregivers. This is particularly important in T2D, where identifying that a patient is at risk of T2D is the first step toward screening and implementing preventive measures. A closer look at the literature shows that, in primary care, we are still falling short when it comes to following diabetes screening recommendations[6] [7] and regularly screening for T2D.[17] This was true even within our own studied population in our 2016 randomized control trial, where the screening rate was 20.4% in the entire population even with a CDSS that contributed to 77.4% of the screened adolescents.[8] This emphasizes the need for systems that, like CHICA, can prioritize more pressing screenings or treatment options while still acting as a link between providers and high-quality evidence.[18] [19]

The finding that more than 50% of CHICA users did not agree that “CHICA makes it easier to get the labs necessary to identify patients who have diabetes or prediabetes” was somewhat surprising given that the rates of screening were significantly increased during the clinical trial and given that specific laboratories were recommended with every prompt. This may be explained by the wording of the question itself since the CHICA system provided instruction on which laboratories to obtain but did not provide a one-click option to order them. To physicians, it is reasonable that the most important actionable step about “getting” the laboratories may be the actual “ordering” part, which CHICA did not facilitate. It may also be explained by the need for extra measures that the providers and staff had to take with CHICA to ensure that the patients actually got the necessary laboratory testing, which might have been perceived as an additional burden. CHICA might also be improving getting laboratories subliminally; however, we still believe that this remains to be a topic of research.

The strengths of this study include a robust computerized CDSS that has been available in the clinics surveyed for over a decade, and therefore users were not just getting used to a new system. Participants answering survey questions had adequate exposure to the CDSS and had used the diabetes CDSS module for at least 12 months. Although there have been studies that assessed the use of CDSS in improving the clinical approach to diabetes,[8] [20] [21] [22] this is one of the few studies that assess the clinician perceptions toward these interventions. Other studies that assessed the provider satisfaction with the use of CDSS in diabetes management have shown findings similar to ours, with providers endorsing CDSS as important.[23] The limitations of this study were the relatively small sample sizes that included only two urban health care centers. These results may not be generalizable to a population of health care providers who have been exposed to another system of CDSS. In addition, we did not have qualitative data to understand the reason behind the lack of endorsement for obtaining laboratories through CHICA. Prospectively, evaluating provider satisfaction over time would validate these findings. An assessment of patient satisfaction when it comes to CHICA implementation for T2D also remains to be a topic for future research.

Conclusion

The CHICA CDSS diabetes-screening module was endorsed by the providers who believed it helped them identify patients who would benefit from screening. Additional work is needed to improve provider and staff satisfaction when it comes to obtaining the right laboratories.

Clinical Relevance Statement

The results of this study provide evidence that despite having a successful CDSS and despite it being endorsed as a successful system for diabetes screening, providers might not necessarily endorse all of this system's screening services.

Multiple Choice Questions

1. Which of the following is a common weakness of assessing user perceptions through surveys with discrete questions and response choices?

   • Difficulty in coding responses for analysis.

   • Inability to assess viewpoints not in the response choices.

   • Fewer responses due to the time and effort involved in gathering data.

   • Respondent confusion in interpreting a rating scale.

   Correct Answer: The correct answer is option b, discrete questions and choices limit the data gathered to assess viewpoints included by the surveyors. Qualitative data gathering (e.g., focus group, open-ended questions) allows for unstructured feedback but may require more time and effort to gather and analyze responses.

2. Which of the following statements regarding CDSS user satisfaction is true?

   • Clinician satisfaction and system usability are essential for the success of a CDSS.

   • If a CDSS has been proven successful, clinicians are always happy to implement and use it.

   • CDSS always disrupts the workflow in the clinics.

   • It is unnecessary to assess the satisfaction of staff members using a CDSS; only providers should be surveyed.

   Correct Answer: The correct answer is option a,: user satisfaction is proven to be necessary for the success of a CDSS. However, successful systems are not always endorsed by all of their users. This emphasizes the need for repeated evaluation of user satisfaction to keep the CDSS users satisfied.

Conflict of Interest

T.S.H. discloses personal fees from Eli Lilly Inc., outside the submitted work. S.M.D. is a creator of the computerized decision support system CHICA that is owned by Indiana University. S.M.D. reports grants from the National Institutes of Health during the conduct of the study. He also is the co-owner of Digital Health Solutions LLC, a company created to make CHICA commercially available. R.W.G. reports grant funding from Digital Health Solutions LLC and Pfizer, outside the submitted work. The rest of the authors have nothing to disclose pertaining to this publication.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. The Indiana University Institutional Review Board approved this study.

• References

• 1 Dabelea D, Mayer-Davis EJ, Saydah S. , et al; SEARCH for Diabetes in Youth Study. Prevalence of type 1 and type 2 diabetes among children and adolescents from 2001 to 2009. JAMA 2014; 311 (17) 1778-1786
• 2 Skinner AC, Skelton JA. Prevalence and trends in obesity and severe obesity among children in the United States, 1999-2012. JAMA Pediatr 2014; 168 (06) 561-566
• 3 Rodbard HW. Diabetes screening, diagnosis, and therapy in pediatric patients with type 2 diabetes. Medscape J Med 2008; 10 (08) 184
• 4 Arslanian S, Bacha F, Grey M, Marcus MD, White NH, Zeitler P. Evaluation and management of youth-onset type 2 diabetes: a position statement by the American Diabetes Association. Diabetes Care 2018; 41 (12) 2648-2668
• 5 American Diabetes Association. 13. Children and Adolescents: Standards of Medical Care in Diabetes-2020 . Diabetes Care 2020; 43 (Suppl. 01) S163-S182
• 6 Lee JM, Eason A, Nelson C, Kazzi NG, Cowan AE, Tarini BA. Screening practices for identifying type 2 diabetes in adolescents. J Adolesc Health 2014; 54 (02) 139-143
• 7 Anand SG, Mehta SD, Adams WG. Diabetes mellitus screening in pediatric primary care. Pediatrics 2006; 118 (05) 1888-1895
• 8 Hannon TS, Dugan TM, Saha CK, McKee SJ, Downs SM, Carroll AE. Effectiveness of computer automation for the diagnosis and management of childhood type 2 diabetes: a randomized clinical trial. JAMA Pediatr 2017; 171 (04) 327-334
• 9 Anand V, Biondich PG, Liu G, Rosenman M, Downs SM. Child Health Improvement through Computer Automation: the CHICA system. Stud Health Technol Inform 2004; 107 (Pt 1): 187-191
• 10 Anand V, Carroll AE, Biondich PG, Dugan TM, Downs SM. Pediatric decision support using adapted Arden Syntax. Artif Intell Med 2018; 92: 15-23
• 11 Carroll AE, Biondich PG, Anand V. , et al. Targeted screening for pediatric conditions with the CHICA system. J Am Med Inform Assoc 2011; 18 (04) 485-490
• 12 Downs SM, Uner H. Expected value prioritization of prompts and reminders. Proc AMIA Symp 2002; 2000: 215-219
• 13 American Diabetes Association. 2. Classification and Diagnosis of Diabetes: Standards of Medical Care in Diabetes-2020 . Diabetes Care 2020; 43 (Suppl. 01) S14-S31
• 14 Grout RW, Cheng ER, Carroll AE, Bauer NS, Downs SM. A six-year repeated evaluation of computerized clinical decision support system user acceptability. Int J Med Inform 2018; 112: 74-81
• 15 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 16 Belamarich PF, Gandica R, Stein REK, Racine AD. Drowning in a sea of advice: pediatricians and American Academy of Pediatrics policy statements. Pediatrics 2006; 118 (04) e964-e978
• 17 Barlow SE, Dietz WH, Klish WJ, Trowbridge FL. Medical evaluation of overweight children and adolescents: reports from pediatricians, pediatric nurse practitioners, and registered dietitians. Pediatrics 2002; 110 (1 Pt 2): 222-228
• 18 O'Connor P. Opportunities to increase the effectiveness of EHR-based diabetes clinical decision support. Appl Clin Inform 2011; 2 (03) 350-354
• 19 O'Connor PJ, Sperl-Hillen JM, Fazio CJ, Averbeck BM, Rank BH, Margolis KL. Outpatient diabetes clinical decision support: current status and future directions. Diabet Med 2016; 33 (06) 734-741
• 20 Ateya MB, Aiyagari R, Moran C, Singer K. Insulin bolus calculator in a pediatric hospital. Safety and user perceptions. Appl Clin Inform 2017; 8 (02) 529-540
• 21 Donsa K, Beck P, Plank J. , et al. A toolbox to improve algorithms for insulin-dosing decision support. Appl Clin Inform 2014; 5 (02) 548-556
• 22 Kantor M, Wright A, Burton M. , et al. Comparison of computer-based clinical decision support systems and content for diabetes mellitus. Appl Clin Inform 2011; 2 (03) 284-303
• 23 Conway N, Adamson KA, Cunningham SG. , et al. Decision support for diabetes in scotland: implementation and evaluation of a clinical decision support system. J Diabetes Sci Technol 2018; 12 (02) 381-388