A Rule-Based Automated Triage Model Using Natural Language Processing for Pain Medicine—Development and Implementation

Abstract

Background

Pain medicine triage plays a crucial role in ensuring patients receive timely and appropriate care by scheduling them to the most suitable treatment path. However, the absence of standardized triage protocols in pain medicine often leads to inefficiencies, including delay of care and wastage of healthcare resources.

Objective

This study aims to develop a rule-based automated referral triage system leveraging information from patients' medical notes for scheduling patients to specific procedures in the pain medicine department.

Methods

The proposed triage system, grounded in the knowledge and expertise of clinical providers, processed referral order comments and referring provider notes by iteratively refining the Natural Language Processing (NLP) rules and post-processing rules through intensively reviewing 76 patients. A post-processing regression model was incorporated to further enhance the accuracy. To ensure alignment with real-world practices, the system was integrated into an electronic health record (EHR) platform for real-time application, streamlining scheduling workflows and enhancing usability in daily clinical settings.

Results

After three iterations, the proposed NLP and post-processing rules improved accuracy from 76.3 to 80.3% compared to machine learning (ML) approaches in the preliminary study. The post-processing model further increased accuracy to 84.2%. The implementation accuracy of 200 cases for the first 3 months was consistent with our prediction at 83.5%, which concluded that the improvement over ML models (p-value = 0.018) was statistically significant at 95% significance level.

Conclusion

This study demonstrates the feasibility and benefits of a knowledge-driven approach to referral triage in specialized medical fields. It lays a foundation for others in building similar triaging solutions to other specialties.

Keywords

pain medicine triage - rule-based model - triage automation - natural language processing - clinical decision support

Protection of Human and Animal Subjects

Our Institutional Review Board (IRB) acknowledges that based on the responses submitted for this study through the Human Subjects Research Wizard tool, and in accordance with the Code of Federal Regulations, 45 CFR 46.102, the study does not require IRB review.

Publikationsverlauf

Eingereicht: 11. April 2025

Angenommen: 04. Dezember 2025

Artikel online veröffentlicht:
18. Dezember 2025

• References

• 1 Goudman L, Rigoard P, Billot M, Duarte RV, Eldabe S, Moens M. Patient selection for spinal cord stimulation in treatment of pain: sequential decision-making model—a narrative review. J Pain Res 2022; 15: 1163-1171
• 2 Jacobson T, Segerberg G. A Machine Learning-based Statistical Analysis of Predictors for Spinal Cord Stimulation Success. Sweden: Umeå University; 2019
• 3 Thomson S, Huygen F, Prangnell S. et al. Applicability and validity of an e-health tool for the appropriate referral and selection of patients with chronic pain for spinal cord stimulation: results from a European retrospective study. Neuromodulation 2023; 26 (01) 164-171
• 4 Pagé MG, Ziemianski D, Shir Y. Triage processes at multidisciplinary chronic pain clinics: an international review of current procedures. Can J Pain 2017; 1 (01) 94-105
• 5 Salman OH, Taha Z, Alsabah MQ, Hussein YS, Mohammed AS, Aal-Nouman M. A review on utilizing machine learning technology in the fields of electronic emergency triage and patient priority systems in telemedicine: coherent taxonomy, motivations, open research challenges and recommendations for intelligent future work. Comput Methods Programs Biomed 2021; 209: 106357
• 6 Jiang H, Mao H, Lu H. et al. Machine learning-based models to support decision-making in emergency department triage for patients with suspected cardiovascular disease. Int J Med Inform 2021; 145: 104326
• 7 Wang B, Li W, Bradlow A, Bazuaye E, Chan AT. Improving triaging from primary care into secondary care using heterogeneous data-driven hybrid machine learning. Decis Support Syst 2023; 166: 113899
• 8 Schöning V, Liakoni E, Baumgartner C. et al. Development and validation of a prognostic COVID-19 severity assessment (COSA) score and machine learning models for patient triage at a tertiary hospital. J Transl Med 2021; 19 (01) 56
• 9 Gottliebsen K, Petersson G. Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review. BMJ Health Care Inform 2020; 27 (01) e100114
• 10 Ilicki J. Challenges in evaluating the accuracy of AI-containing digital triage systems: a systematic review. PLoS One 2022; 17 (12) e0279636
• 11 Char DS, Abràmoff MD, Feudtner C. Identifying ethical considerations for machine learning healthcare applications. Am J Bioeth 2020; 20 (11) 7-17
• 12 Jiang L, Huang YL, Fan J, Hunt CL, Eldrige JS. Development and implementation of automated referral triaging system for spinal cord stimulation procedure in pain medicine. J Med Syst 2025; 49 (01) 14
• 13 Berner ES, La Lande TJ. Overview of clinical decision support systems. In: Berner E, ed. Clinical Decision Support Systems: Theory and Practice. Springer; , Cham; 2016: 1-7
• 14 Aziz A, Rodriguez S, Chatwin C. From guidelines to practice: improving clinical care through rule-based clinical decision support at the point of care. In: Leung TY, Métais E, eds. Rules on the Web. From Theory to Applications: 8th International Symposium, RuleML 2014, Co-located with the 21st European Conference on Artificial Intelligence, ECAI 2014, Prague, Czech Republic, August 18–20, 2014. Proceedings. Vol 8. Springer International Publishing; 2014: 178-185
• 15 Leder Macek AJ, Kirschenbaum JD, Ricklan SJ, Schreiber-Stainthorp W, Omene BC, Conderino S. Validation of rule-based algorithms to determine colorectal, breast, and cervical cancer screening status using electronic health record data from an urban healthcare system in New York City. Prev Med Rep 2021; 24: 101599
• 16 Gebremariam BM, Aboye GT, Dessalegn AA, Simegn GL. Rule-based expert system for the diagnosis of maternal complications during pregnancy: for low resource settings. Digit Health 2024 10. :20552076241230073
• 17 Çelik Ertuğrul D, Toygar Ö, Foroutan N. A rule-based decision support system for aiding iron deficiency management. Health Informatics J 2021 27. 04 ):14604582211066054
• 18 Papadopoulos P, Soflano M, Chaudy Y, Adejo W, Connolly TM. A systematic review of technologies and standards used in the development of rule-based clinical decision support systems. Health Technol (Berl) 2022; 12 (04) 713-727
• 19 Wen A, Fu S, Moon S. et al. Desiderata for delivering NLP to accelerate healthcare AI advancement and a Mayo Clinic NLP-as-a-service implementation. NPJ Digit Med 2019; 2 (01) 130
• 20 Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform 2009; 42 (05) 839-851
• 21 Westra BL, Johnson SG, Ali S. et al. Validation and refinement of a pain information model from EHR flowsheet data. Appl Clin Inform 2018; 9 (01) 185-198
• 22 Harle CA, DiIulio J, Downs SM. et al. Decision-centered design of patient information visualizations to support chronic pain care. Appl Clin Inform 2019; 10 (04) 719-728
• 23 CColeman B, Finch D, Wang R. et al. Extracting pain care quality indicators from U.S. Veterans Health Administration chiropractic care using natural language processing. Appl Clin Inform 2023; 14 (03) 600-608