Download PDF
Methods Inf Med 2022; 61(05/06): 174-184
DOI: 10.1055/a-1905-5639
Original Article

DxGenerator: An Improved Differential Diagnosis Generator for Primary Care Based on MetaMap and Semantic Reasoning

Ali Sanaeifar
1   Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran
,
Saeid Eslami
1   Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran
,
Mitra Ahadi
2   Department of Gastroenterology and Hepatology, School of Medicine, Mashhad University of Medical Sciences, MUMS, Iran
,
Mohsen Kahani
3   Department of Computer Engineering, Ferdowsi University of Mashhad, Mashhad, Iran
,
Hassan Vakili Arki
1   Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran
› Author Affiliations
› Further Information
    Permissions and Reprints

Abstract

Background In recent years, researchers have used many computerized interventions to reduce medical errors, the third cause of death in developed countries. One of such interventions is using differential diagnosis generators in primary care, where physicians may encounter initial symptoms without any diagnostic presuppositions. These systems generate multiple diagnoses, ranked by their likelihood. As such, these reports' accuracy can be determined by the location of the correct diagnosis in the list.

Objective This study aimed to design and evaluate a novel practical web-based differential diagnosis generator solution in primary care.

Methods In this research, a new online clinical decision support system, called DxGenerator, was designed to improve diagnostic accuracy; to this end, an attempt was made to converge a semantic database with the unified medical language system (UMLS) knowledge base, using MetaMap tool and natural language processing. In this regard, 120 diseases of gastrointestinal organs causing abdominal pain were modeled into the database. After designing an inference engine and a pseudo-free-text interactive interface, 172 patient vignettes were inputted into DxGenerator and ISABEL, the most accurate similar system. The Wilcoxon signed ranked test was used to compare the position of correct diagnoses in DxGenerator and ISABEL. The α level was defined as 0.05.

Results On a total of 172 vignettes, the mean and standard deviation of correct diagnosis positions improved from 4.2 ± 5.3 in ISABEL to 3.2 ± 3.9 in DxGenerator. This improvement was significant in the subgroup of uncommon diseases (p-value < 0.05).

Conclusion Using UMLS knowledge base and MetaMap Tools can improve the accuracy of diagnostic systems in which terms are entered in a free text manner. Applying these new methods will help the medical community accept medical diagnostic systems better.

Keywords

differential diagnosis - clinical decision support systems - unified medical language system - MetaMap - natural language processing

Protection of Human and Animal Subjects

No patients were included in this study. This study used patient vignettes to evaluate the proposed system.


Supplementary Material



Publication History

Received: 10 March 2022

Accepted: 17 July 2022

Accepted Manuscript online:
20 July 2022

Article published online:
17 November 2022

© 2022. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Singh H, Schiff GD, Graber ML, Onakpoya I, Thompson MJ. The global burden of diagnostic errors in primary care. BMJ Qual Saf 2017; 26 (06) 484-494
    CrossrefPubMedGoogle Scholar
  • 2 Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ 2016; 353: i2139
    CrossrefPubMedGoogle Scholar
  • 3 Donaldson MS, Corrigan JM, Kohn LT. To Err is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000
    Google Scholar
  • 4 Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med 2008; 121 (5, Suppl): S2-S23
    CrossrefPubMedGoogle Scholar
  • 5 Riches N, Panagioti M, Alam R. et al. The effectiveness of electronic differential diagnoses (DDX) generators: a systematic review and meta-analysis. PLoS One 2016; 11 (03) e0148991
    CrossrefPubMedGoogle Scholar
  • 6 Singh H, Giardina TD, Meyer AN, Forjuoh SN, Reis MD, Thomas EJ. Types and origins of diagnostic errors in primary care settings. JAMA Intern Med 2013; 173 (06) 418-425
    CrossrefPubMedGoogle Scholar
  • 7 Minué S, Bermúdez-Tamayo C, Fernández A. et al. Identification of factors associated with diagnostic error in primary care. BMC Fam Pract 2014; 15 (01) 92
    CrossrefPubMedGoogle Scholar
  • 8 Balla J, Heneghan C, Goyder C, Thompson M. Identifying early warning signs for diagnostic errors in primary care: a qualitative study. BMJ Open 2012; 2 (05) e001539
    CrossrefPubMedGoogle Scholar
  • 9 Aronson AR, Lang F-M. An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc 2010; 17 (03) 229-236
    CrossrefPubMedGoogle Scholar
  • 10 Reátegui R, Ratté S. Comparison of MetaMap and cTAKES for entity extraction in clinical notes. BMC Med Inform Decis Mak 2018; 18 (Suppl. 03) 7-4
    PubMedGoogle Scholar
  • 11 Chiaramello E, Paglialonga A, Pinciroli F, Tognola G. Attempting to use MetaMap in clinical practice: a feasibility study on the identification of medical concepts from Italian clinical notes. Stud Health Technol Inform 2016; 228: 28-32
    PubMedGoogle Scholar
  • 12 Berner ES, Maisiak RS, Cobbs CG, Taunton OD. Effects of a decision support system on physicians' diagnostic performance. J Am Med Inform Assoc 1999; 6 (05) 420-427
    CrossrefPubMedGoogle Scholar
  • 13 Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med 2003; 78 (08) 775-780
    CrossrefPubMedGoogle Scholar
  • 14 Kostopoulou O, Oudhoff J, Nath R. et al. Predictors of diagnostic accuracy and safe management in difficult diagnostic problems in family medicine. Med Decis Making 2008; 28 (05) 668-680
    CrossrefPubMedGoogle Scholar
  • 15 Kostopoulou O, Lionis C, Angelaki A, Ayis S, Durbaba S, Delaney BC. Early diagnostic suggestions improve accuracy of family physicians: a randomized controlled trial in Greece. Fam Pract 2015; 32 (03) 323-328
    CrossrefPubMedGoogle Scholar
  • 16 Bösner S, Pickert J, Stibane T. Teaching differential diagnosis in primary care using an inverted classroom approach: student satisfaction and gain in skills and knowledge. BMC Med Educ 2015; 15 (01) 63
    CrossrefPubMedGoogle Scholar
  • 17 McParland CR, Cooper MA, Johnston B. Differential diagnosis decision support systems in primary and out-of-hours care: a qualitative analysis of the needs of key stakeholders in Scotland. J Prim Care Community Health 2019; 10: 2150132719829315
    CrossrefPubMedGoogle Scholar
  • 18 Porat T, Delaney B, Kostopoulou O. The impact of a diagnostic decision support system on the consultation: perceptions of GPs and patients. BMC Med Inform Decis Mak 2017; 17 (01) 79
    CrossrefPubMedGoogle Scholar
  • 19 Kawamura R, Harada Y, Sugimoto S, Nagase Y, Katsukura S, Shimizu T. Incidence of diagnostic errors among unexpectedly hospitalized patients using an automated medical history-taking system with a differential diagnosis generator: retrospective observational study. JMIR Med Inform 2022; 10 (01) e35225
    CrossrefPubMedGoogle Scholar
  • 20 Bond WF, Schwartz LM, Weaver KR, Levick D, Giuliano M, Graber ML. Differential diagnosis generators: an evaluation of currently available computer programs. J Gen Intern Med 2012; 27 (02) 213-219
    CrossrefPubMedGoogle Scholar
  • 21 Garg AX, Adhikari NK, McDonald H. et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
    CrossrefPubMedGoogle Scholar
  • 22 Gerbert B, Bronstone A, Maurer T, Hofmann R, Berger T. Decision support software to help primary care physicians triage skin cancer: a pilot study. Arch Dermatol 2000; 136 (02) 187-192
    CrossrefPubMedGoogle Scholar
  • 23 Henderson EJ, Rubin GP. The utility of an online diagnostic decision support system (Isabel) in general practice: a process evaluation. JRSM Short Rep 2013; 4 (05) 31
    CrossrefPubMedGoogle Scholar
  • 24 Bond W, Schwartz L, Weaver K, Levick D, Giuliano M, Graber M. A qualitative review of differential diagnosis generators. Poster presented at: The 32 nd Annual Meeting of the Society for Medical Decision Making; 2010. Ontario, Canada:
    Google Scholar
  • 25 Lemaire JB, Schaefer JP, Martin LA, Faris P, Ainslie MD, Hull RD. Effectiveness of the quick medical reference as a diagnostic tool. CMAJ 1999; 161 (06) 725-728
    PubMedGoogle Scholar
  • 26 Nelson SJ, Blois MS, Tuttle MS. et al. Evaluating RECONSIDER. A computer program for diagnostic prompting. J Med Syst 1985; 9 (5-6): 379-388
    PubMedGoogle Scholar
  • 27 Miller RA, Pople Jr HE, Myers JD. Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med 1982; 307 (08) 468-476
    CrossrefPubMedGoogle Scholar
  • 28 De Dombal FT, Leaper DJ, Horrocks JC, Staniland JR, McCann AP. Human and computer-aided diagnosis of abdominal pain: further report with emphasis on performance of clinicians. BMJ 1974; 1 (5904): 376-380
    CrossrefPubMedGoogle Scholar
  • 29 Feldman MJ, Barnett GO. An approach to evaluating the accuracy of DXplain. Comput Methods Programs Biomed 1991; 35 (04) 261-266
    CrossrefPubMedGoogle Scholar
  • 30 Mohammed O, Benlamri R, Fong S. Building A Diseases Symptoms Ontology for Medical Diagnosis: An Integrative Approach. Paper presented at: First International Conference on Future Generation Communication Technologies 2012. London, UK: IEEE; 2012: 104-108
    Google Scholar
  • 31 Schriml LM, Arze C, Nadendla S. et al. Disease ontology: a backbone for disease semantic integration. Nucleic Acids Res 2012; 40 (Database issue): D940-D946
    CrossrefPubMedGoogle Scholar
  • 32 de Lusignan S, Liaw S-T, Michalakidis G, Jones S. Defining datasets and creating data dictionaries for quality improvement and research in chronic disease using routinely collected data: an ontology-driven approach. Inform Prim Care 2011; 19 (03) 127-134
    PubMedGoogle Scholar
  • 33 García-Crespo Á, Rodríguez A, Mencke M, Gómez-Berbís JM, Colomo-Palacios R. ODDIN: Ontology-driven differential diagnosis based on logical inference and probabilistic refinements. Expert Syst Appl 2010; 37 (03) 2621-2628
    CrossrefPubMedGoogle Scholar
  • 34 Riaño D, Real F, López-Vallverdú JA. et al. An ontology-based personalization of health-care knowledge to support clinical decisions for chronically ill patients. J Biomed Inform 2012; 45 (03) 429-446
    CrossrefPubMedGoogle Scholar
  • 35 Valls A, Gibert K, Sánchez D, Batet M. Using ontologies for structuring organizational knowledge in Home Care assistance. Int J Med Inform 2010; 79 (05) 370-387
    CrossrefPubMedGoogle Scholar
  • 36 Sanaeifar A, Faraahi A, Mahmood T. SEPHYRES 1: a symptom checker based on semantic pain descriptors and weight spreading. Appl Med Inform 2016; 38 (3–4): 105-116
    PubMedGoogle Scholar
  • 37 Sanaeifar A, Tara M, Faraahi A, Mousavi BM, Ahadi M, Bahari A. SEPHYRES 2: applying semantic-pseudo-fuzzy methods in medical diagnostic ontologies. Int J Med Eng Inform 2018; 10 (03) 265-278
    PubMedGoogle Scholar
  • 38 Naderi H, Madani S, Kiani B, Etminani K. Similarity of medical concepts in question and answering of health communities. Health Informatics J 2020; 26 (02) 1443-1454
    CrossrefPubMedGoogle Scholar
  • 39 Naderi H, Kiani B, Madani S, Etminani K. Concept based auto-assignment of healthcare questions to domain experts in online Q&A communities. Int J Med Inform 2020; 137: 104108
    CrossrefPubMedGoogle Scholar
  • 40 Kang T, Perotte A, Tang Y, Ta C, Weng C. UMLS-based data augmentation for natural language processing of clinical research literature. J Am Med Inform Assoc 2021; 28 (04) 812-823
    CrossrefPubMedGoogle Scholar
  • 41 Michalopoulos G, Wang Y, Kaka H, Chen H, Wong A. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. Presented at: Proceeding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; 2021: 1744-1753
    PubMedGoogle Scholar
  • 42 Natural Language Toolkit. . Accessed Jun 20, 2022 at: https://www.nltk.org
    PubMed
  • 43 Python Wrapper for MetaMap. . Accessed Jun 20, 2022 at: https://github.com/smujjiga/pymm
    PubMed
  • 44 Artificial Intelligence - Expert Systems. . Accessed Jun 20, 2022 at: https://www.tutorialspoint.com/artificial_intelligence/artificial_intelligence_expert_systems.htm
    PubMed
  • 45 Fauci AS. Harrison's Principles of Internal Medicine. 20th edition. New York, USA: McGraw-Hill Education; 2015
    Google Scholar
  • 46 Benjamin IJ, Griggs RC, Wing EJ, Fitz JG. Andreoli and Carpenter's Cecil Essentials of Medicine. 9th edition. Chicago US: Saunders: 2016
    Google Scholar
  • 47 UpToDate. 2020 . Accessed Jun 20, 2022 at: https://Uptodate.com
    PubMed
  • 48 Similarity Functions. . Accessed Jun 20, 2022 at: https://neo4j.com/docs/graph-data-science/current/alpha-algorithms/jaccard/
    PubMed
  • 49 Edit Distance and Jaccard Distance Calculation with NLTK. Accessed Jun 20, 2022 at: https://python.gotrained.com/nltk-edit-distance-jaccard-distance/#Jaccard_Distance
    PubMedGoogle Scholar
  • 50 Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 2015; 351: h3480
    CrossrefPubMedGoogle Scholar
  • 51 Kostopoulou O, Porat T, Corrigan D, Mahmoud S, Delaney BC. Diagnostic accuracy of GPs when using an early-intervention decision support system: a high-fidelity simulation. Br J Gen Pract 2017; 67 (656) e201-e208
    CrossrefPubMedGoogle Scholar