Design, Implementation, and Validation of an Automated, Algorithmic COVID-19 Triage Tool

Abstract

Objective We describe the design, implementation, and validation of an online, publicly available tool to algorithmically triage patients experiencing severe acute respiratory syndrome coronavirus (SARS-CoV-2)-like symptoms.

Methods We conducted a chart review of patients who completed the triage tool and subsequently contacted our institution's phone triage hotline to assess tool- and clinician-assigned triage codes, patient demographics, SARS-CoV-2 (COVID-19) test data, and health care utilization in the 30 days post-encounter. We calculated the percentage of concordance between tool- and clinician-assigned triage categories, down-triage (clinician assigning a less severe category than the triage tool), and up-triage (clinician assigning a more severe category than the triage tool) instances.

Results From May 4, 2020 through January 31, 2021, the triage tool was completed 30,321 times by 20,930 unique patients. Of those 30,321 triage tool completions, 51.7% were assessed by the triage tool to be asymptomatic, 15.6% low severity, 21.7% moderate severity, and 11.0% high severity. The concordance rate, where the triage tool and clinician assigned the same clinical severity, was 29.2%. The down-triage rate was 70.1%. Only six patients were up-triaged by the clinician. 72.1% received a COVID-19 test administered by our health care system within 14 days of their encounter, with a positivity rate of 14.7%.

Conclusion The design, pilot, and validation analysis in this study show that this COVID-19 triage tool can safely triage patients when compared with clinician triage personnel. This work may signal opportunities for automated triage of patients for conditions beyond COVID-19 to improve patient experience by enabling self-service, on-demand, 24/7 triage access.

Keywords

clinical decision support - human–computer interaction - improvement - automated triage - algorithmic medicine

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. This study was reviewed by University of Pennsylvania Institutional Review Board and determined to be exempt as it met criteria for IRB review exemption authorized by 45 CFR 46.104, category 4.

^* Equal co-first author contribution.

Publication History

Received: 06 July 2021

Accepted: 17 September 2021

Article published online:
03 November 2021

• References

• 1 Gann B. Giving patients choice and control: health informatics on the patient journey. Yearb Med Inform 2012; 7: 70-73
• 2 Fox S, Duggan M. Internet and American life project. Pew Research Center and California Health Care Foundation, 2013:4. Google Search. Health Online. Published 2013. Accessed April 30, 2021 at: https://www.pewresearch.org/internet/2013/01/15/health-online-2013/
• 3 North F, Ward WJ, Varkey P, Tulledge-Scheitel SM. Should you search the Internet for information about your acute symptom?. Telemed J E Health 2012; 18 (03) 213-218
• 4 Black P. The dangers of using Google as a diagnostic aid. Br J Nurs 2009; 18 (19) 1157
• 5 Luger TM, Houston TK, Suls J. Older adult experience of online diagnosis: results from a scenario-based think-aloud protocol. J Med Internet Res 2014; 16 (01) e16
• 6 Chen Z, Turner MR. The internet for self-diagnosis and prognostication in ALS. Amyotroph Lateral Scler 2010; 11 (06) 565-567
• 7 Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 2015; 351: h3480
• 8 Knighton AJ, Ranade-Kharkar P, Brunisholz KD. et al. Rapid implementation of a complex, multimodal technology response to COVID-19 at an integrated community-based health care system. Appl Clin Inform 2020; 11 (05) 825-838
• 9 Grange ES, Neil EJ, Stoffel M. et al. Responding to COVID-19: the UW medicine information technology services experience. Appl Clin Inform 2020; 11 (02) 265-275
• 10 White-Dzuro CG, Schultz JD, Ye C. et al. Extracting medical information from paper COVID-19 assessment forms. Appl Clin Inform 2021; 12 (01) 170-178
• 11 Herriman M, Meer E, Rosin R, Lee V, Washington V, Volpp KG. Asked and Answered: Building a Chatbot to Address COVID-19-Related Concerns. NEJM Catalyst. Published 2020. Available at: https://catalyst.nejm.org/doi/full/10.1056/cat.20.0230
• 12 Facts and Figures. 2021. Penn Medicine. Accessed April 30, 2021 at: https://spark.adobe.com/page/d6tXWsvs6Cy5j/
• 13 (No Title). Accessed April 30, 2021 at: https://www.pennmedicine.org/news/-/media/documentsandaudio/prnews/factsand figures/pennmedicinefacts2012.ashx
• 14 Judson TJ, Odisho AY, Neinstein AB. et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc 2020; 27 (06) 860-866
• 15 Galinato J, Montie M, Patak L, Titler M. Perspectives of nurses and patients on call light technology. Comput Inform Nurs 2015; 33 (08) 359-367
• 16 Philadelphia Metro Area - Profile data - Census Reporter. Accessed April 30, 2021 at: https://censusreporter.org/profiles/31000US37980-philadelphia-camden-wilmington-pa-nj-de-md-metro-area/
• 17 Powley L, McIlroy G, Simons G, Raza K. Are online symptoms checkers useful for patients with inflammatory arthritis?. BMC Musculoskelet Disord 2016; 17 (01) 362
• 18 Dickerson S, Reinhart AM, Feeley TH. et al. Patient internet use for health information at three urban primary care clinics. J Am Med Inform Assoc 2004; 11 (06) 499-504