Download PDF
Appl Clin Inform 2023; 14(05): 932-943
DOI: 10.1055/a-2184-6481
Research Article

Effectiveness of a Vendor Predictive Model for the Risk of Pediatric Asthma Exacerbation: A Difference-in-Differences Analysis

Avinash Murugan
1   Department of Medicine, Yale New Haven Hospital, New Haven, Connecticut, United States
,
Swaminathan Kandaswamy
2   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
,
Edwin Ray
3   Information Services and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
,
Scott Gillespie
2   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
,
Evan Orenstein
2   Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia, United States
3   Information Services and Technology, Children's Healthcare of Atlanta, Atlanta, Georgia, United States
› Author Affiliations
› Further Information
    Permissions and Reprints

Abstract

Background Asthma is a common cause of morbidity and mortality in children. Predictive models may help providers tailor asthma therapies to an individual's exacerbation risk. The effectiveness of asthma risk scores on provider behavior and pediatric asthma outcomes remains unknown.

Objective Determine the impact of an electronic health record (EHR) vendor-released model on outcomes for children with asthma.

Methods The Epic Systems Risk of Pediatric Asthma Exacerbation model was implemented on February 24, 2021, for volunteer pediatric allergy and pulmonology providers as a noninterruptive risk score visible in the patient schedule view. Asthma hospitalizations, emergency department (ED) visits, or oral steroid courses within 90 days of the index visit were compared from February 24, 2019, to February 23, 2022, using a difference-in-differences design with a control group of visits to providers in the same departments. Volunteer providers were interviewed to identify barriers and facilitators to model use.

Results In the intervention group, asthma hospitalizations within 90 days decreased from 1.4% (54/3,842) to 0.7% (14/2,165) after implementation with no significant change in the control group (0.9% [171/19,865] preimplementation to 1.0% [105/10,743] post). ED visits in the intervention group decreased from 5.8% (222/3,842) to 5.5% (118/2,164) but increased from 5.5% (1,099/19,865) to 6.8% (727/10,743) in the control group. The adjusted difference-in-differences estimators for hospitalization, ED visit, and oral steroid outcomes were −0.9% (95% confidence interval [CI]: −1.6 to −0.3), –2.4% (−3.9 to −0.8), and –1.9% (−4.3 to 0.5). In qualitative analysis, providers understood the purpose of the model and felt it was useful to flag high exacerbation risk. Trust in the model was calibrated against providers' own clinical judgement.

Conclusion This EHR vendor model implementation was associated with a significant decrease in asthma hospitalization and ED visits within 90 days of pediatric allergy and pulmonology clinic visits, but not oral steroid courses.

Keywords

electronic health record - asthma - clinical decision support

Protection of Human and Animal Subjects

This project was deemed to be nonhuman subjects research as a quality improvement initiative by the Children's Healthcare of Atlanta IRB (identifier: STUDY00000905).


Authors' Contributions

A.M. and E.O. conceptualized the study design.


A.M., S.K., E.R., and E.O. implemented the study.


A.M., S.G., and E.O. designed and performed the statistical analysis.


A.M., S.K., E.R., S.G., and E.O. wrote and edited the manuscript.




Publication History

Received: 13 June 2023

Accepted: 28 September 2023

Accepted Manuscript online:
29 September 2023

Article published online:
29 November 2023

© 2023. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Ferrante G, La Grutta S. The burden of pediatric asthma. Front Pediatr 2018; 6: 186
    CrossrefPubMedGoogle Scholar
  • 2 Perry R, Braileanu G, Palmer T, Stevens P. The economic burden of pediatric asthma in the United States: literature review of current evidence. PharmacoEconomics 2019; 37 (02) 155-167
    CrossrefPubMedGoogle Scholar
  • 3 Hughes HK, Matsui EC, Tschudy MM, Pollack CE, Keet CA. Pediatric asthma health disparities: race, hardship, housing, and asthma in a national survey. Acad Pediatr 2017; 17 (02) 127-134
    CrossrefPubMedGoogle Scholar
  • 4 Avery C, Perrin EM, Lang JE. Updates to the pediatrics asthma management guidelines. JAMA Pediatr 2021; 175 (09) 966-967
    CrossrefPubMedGoogle Scholar
  • 5 Schuler M, Faller H, Wittmann M, Schultz K. Asthma Control Test and Asthma Control Questionnaire: factorial validity, reliability and correspondence in assessing status and change in asthma control. J Asthma 2016; 53 (04) 438-445
    CrossrefPubMedGoogle Scholar
  • 6 Jia CE, Zhang HP, Lv Y. et al. The Asthma Control Test and Asthma Control Questionnaire for assessing asthma control: systematic review and meta-analysis. J Allergy Clin Immunol 2013; 131 (03) 695-703
    CrossrefPubMedGoogle Scholar
  • 7 Tsai CL, Clark S, Camargo Jr CA. Risk stratification for hospitalization in acute asthma: the CHOP classification tree. Am J Emerg Med 2010; 28 (07) 803-808
    CrossrefPubMedGoogle Scholar
  • 8 Lieu TA, Quesenberry CP, Sorel ME, Mendoza GR, Leong AB. Computer-based models to identify high-risk children with asthma. Am J Respir Crit Care Med 1998; 157 (4, Pt 1): 1173-1180
    CrossrefPubMedGoogle Scholar
  • 9 Hanson JR, Lee BR, Williams DD. et al. Developing a risk stratification model for predicting future health care use in asthmatic children. Ann Allergy Asthma Immunol 2016; 116 (01) 26-30
    CrossrefPubMedGoogle Scholar
  • 10 Martin A, Bauer V, Datta A. et al. Development and validation of an asthma exacerbation prediction model using electronic health record (EHR) data. J Asthma 2020; 57 (12) 1339-1346
    CrossrefPubMedGoogle Scholar
  • 11 Niu C, Xu Y, Schuler CL. et al. Evaluation of risk scores to predict pediatric severe asthma exacerbations. J Allergy Clin Immunol Pract 2021; 9 (12) 4393-4401.e8
    CrossrefPubMedGoogle Scholar
  • 12 Patel D, Hall GL, Broadhurst D, Smith A, Schultz A, Foong RE. Does machine learning have a role in the prediction of asthma in children?. Paediatr Respir Rev 2022; 41: 51-60
    PubMedGoogle Scholar
  • 13 Seol HY, Shrestha P, Muth JF. et al. Artificial intelligence-assisted clinical decision support for childhood asthma management: a randomized clinical trial. PLoS One 2021; 16 (08) e0255261
    CrossrefPubMedGoogle Scholar
  • 14 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018; 178 (11) 1544-1547
    CrossrefPubMedGoogle Scholar
  • 15 Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366 (6464): 447-453
    CrossrefPubMedGoogle Scholar
  • 16 Wong A, Otles E, Donnelly JP. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. [published correction appears in JAMA Intern Med. 2021 Aug 1;181(8):1144] JAMA Intern Med 2021; 181 (08) 1065-1070
    CrossrefPubMedGoogle Scholar
  • 17 Lu JH, Callahan A, Patel BS, Morse KE, Dash D, Pfeffer MA, Shah NH. Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor: A Systematic Review. JAMA Netw Open 2022; 5 (08) e2227779
    CrossrefPubMedGoogle Scholar
  • 18 Galaxy - cognitive computing model brief: risk of pediatric asthma exacerbation [Internet]. Accessed November 15, 2022 at: https://galaxy.epic.com/?#Browse/page=1!68!95!100041319
    PubMed
  • 19 Colonescu C. 7.7 The difference-in-differences estimator [Internet]. In: Principles of Econometrics with R. 2016. Accessed November 15, 2022 at: https://bookdown.org/ccolonescu/RPoE4/
    PubMed
  • 20 Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4: 50
    CrossrefPubMedGoogle Scholar
  • 21 Mcdermott P, Dominguez C, Kasdaglis N, Ryan M, Trahan I, Nelson A. Human-machine teaming systems engineering guide | MITRE [Internet]. 2018. Accessed November 15, 2022 at: https://www.mitre.org/news-insights/publication/human-machine-teaming-systems-engineering-guide
    PubMed
  • 22 Major VJ, Jones SA, Razavian N. et al. Evaluating the effect of a COVID-19 predictive model to facilitate discharge: a randomized controlled trial. Appl Clin Inform 2022; 13 (03) 632-640
    Article in Thieme ConnectPubMedGoogle Scholar
  • 23 Wu AC, Tantisira K, Li L, Schuemann B, Weiss ST, Fuhlbrigge AL. Childhood Asthma Management Program Research Group. Predictors of symptoms are different from predictors of severe exacerbations from asthma in children. Chest 2011; 140 (01) 100-107
    CrossrefPubMedGoogle Scholar
  • 24 Suruki RY, Daugherty JB, Boudiaf N, Albers FC. The frequency of asthma exacerbations and healthcare utilization in patients with asthma from the UK and USA. BMC Pulm Med 2017; 17 (01) 74
    CrossrefPubMedGoogle Scholar
  • 25 Correa-Agudelo E, Ding L, Beck AF. et al. Understanding racial disparities in childhood asthma using individual- and neighborhood-level risk factors. J Allergy Clin Immunol 2022; 150 (06) 1427-1436.e5
    CrossrefPubMedGoogle Scholar
  • 26 Kaufmann J, Marino M, Lucas J. et al. Racial and ethnic disparities in acute care use for pediatric asthma. Ann Fam Med 2022; 20 (02) 116-122
    CrossrefPubMedGoogle Scholar
  • 27 Schatz M, Clark S, Emond JA, Schreiber D, Camargo Jr CA. Sex differences among children 2-13 years of age presenting at the emergency department with acute asthma. Pediatr Pulmonol 2004; 37 (06) 523-529
    CrossrefPubMedGoogle Scholar
  • 28 Plombon S, S Rudin R, Sulca Flores J. et al. Assessing equitable recruitment in a digital health trial for asthma. Appl Clin Inform 2023; 14 (04) 620-631
    Article in Thieme ConnectPubMedGoogle Scholar
  • 29 Gao E, Radparvar I, Dieu H, Ross MK. User experience design for adoption of asthma clinical decision support tools. Appl Clin Inform 2022; 13 (04) 971-982
    Article in Thieme ConnectPubMedGoogle Scholar
  • 30 Tarabichi Y, Cheng A, Bar-Shain D. et al. Improving timeliness of antibiotic administration using a provider and pharmacist facing sepsis early warning system in the emergency department setting: a randomized controlled quality improvement initiative. Crit Care Med 2022; 50 (03) 418-427
    CrossrefPubMedGoogle Scholar
  • 31 Li RC, Asch SM, Shah NH. Developing a delivery science for artificial intelligence in healthcare. NPJ Digit Med 2020; 3: 107
    CrossrefPubMedGoogle Scholar