Automated Classification of Free-Text Radiology Reports: Using Different Feature Extraction Methods to Identify Fractures of the Distal Fibula

Abstract

Purpose Radiology reports mostly contain free-text, which makes it challenging to obtain structured data. Natural language processing (NLP) techniques transform free-text reports into machine-readable document vectors that are important for creating reliable, scalable methods for data analysis. The aim of this study is to classify unstructured radiograph reports according to fractures of the distal fibula and to find the best text mining method.

Materials & Methods We established a novel German language report dataset: a designated search engine was used to identify radiographs of the ankle and the reports were manually labeled according to fractures of the distal fibula. This data was used to establish a machine learning pipeline, which implemented the text representation methods bag-of-words (BOW), term frequency-inverse document frequency (TF-IDF), principal component analysis (PCA), non-negative matrix factorization (NMF), latent Dirichlet allocation (LDA), and document embedding (doc2vec). The extracted document vectors were used to train neural networks (NN), support vector machines (SVM), and logistic regression (LR) to recognize distal fibula fractures. The results were compared via cross-tabulations of the accuracy (acc) and area under the curve (AUC).

Results In total, 3268 radiograph reports were included, of which 1076 described a fracture of the distal fibula. Comparison of the text representation methods showed that BOW achieved the best results (AUC = 0.98; acc = 0.97), followed by TF-IDF (AUC = 0.97; acc = 0.96), NMF (AUC = 0.93; acc = 0.92), PCA (AUC = 0.92; acc = 0.9), LDA (AUC = 0.91; acc = 0.89) and doc2vec (AUC = 0.9; acc = 0.88). When comparing the different classifiers, NN (AUC = 0,91) proved to be superior to SVM (AUC = 0,87) and LR (AUC = 0,85).

Conclusion An automated classification of unstructured reports of radiographs of the ankle can reliably detect findings of fractures of the distal fibula. A particularly suitable feature extraction method is the BOW model.

Key Points: 

• The aim was to classify unstructured radiograph reports according to distal fibula fractures.

• Our automated classification system can reliably detect fractures of the distal fibula.

• A particularly suitable feature extraction method is the BOW model.

Citation Format

• Dewald CL, Balandis A, Becker LS et al. Automated Classification of Free-Text Radiology Reports: Using Different Feature Extraction Methods to Identify Fractures of the Distal Fibula. Fortschr Röntgenstr 2023; 195: 713 – 719

Zusammenfassung

Ziel Radiologische Befundtexte enthalten häufig Freitext, was eine strukturierte Datenauswertung erschwert. Natural language processing (NLP)-Techniken wandeln Freitext in maschinenlesbare Dokumentenvektoren um, die für die Entwicklung zuverlässiger, skalierbarer Methoden zur Datenanalyse wichtig sind. Ziel dieser Studie war es, unstrukturierte Röntgenbefunde nach Frakturen der distalen Fibula zu klassifizieren und die beste Text-Mining-Methode zu finden.

Material & Methoden Zur Erstellung eines eigenen deutschsprachigen Befunddatensatzes wurden mittels einer dedizierten Suchmaschine Sprunggelenks-Röntgenbilder identifiziert und die entsprechenden Befunde manuell nach Frakturen der distalen Fibula sortiert. Anhand der Daten wurde eine Machine-Learning-Pipeline erstellt, die die Textrepräsentationsmethoden Bag-of-Words (BOW), Term Frequency-Inverse Document Frequency (TF-IDF), Principal Component Analysis (PCA), Non-Negative Matrix Factorization (NMF), Latent Dirichlet Allocation (LDA) und Document Embedding (doc2vec) implementierte. Die extrahierten Dokumentvektoren wurden zum Trainieren von neuronalen Netzen (NN), Support Vector Machines (SVM) und logistischer Regression (LR) verwendet, um distale Fibulafrakturen zu erkennen. Die Ergebnisse wurden mittels Kreuztabellen bzgl. der Accuracy (acc) und der area under the curve (AUC) verglichen.

Ergebnisse Insgesamt wurden 3268 Röntgenbefunde inkludiert, von denen 1076 eine distale Fibulafraktur beschrieben. Der Vergleich der Textdarstellungsmethoden zeigte, dass BOW die besten Ergebnisse erzielte (AUC = 0,98; acc = 0,97), gefolgt von TF-IDF (AUC = 0,97; acc = 0,96), NMF (AUC = 0,93; acc = 0,92), PCA (AUC = 0,92; acc = 0,9), LDA (AUC = 0,91; acc = 0,89) und doc2vec (AUC = 0,9; acc = 0,88). Im Vergleich der Klassifikatoren erwiesen sich die NN (AUC = 0,91) gegenüber SVM (AUC = 0,87) und LR (AUC = 0,85) als überlegen.

Schlussfolgerung Durch die automatisierte Klassifikation von unstrukturierten Befunden von Sprunggelenksaufnahmen können Frakturen der distalen Fibula zuverlässig erkannt werden. Eine besonders geeignete Methode zur Feature Extraction ist das BOW-Modell.

Kernaussagen: 

• Ziel war die automatisierte Klassifizierung unstrukturierter Röntgenbefunde entsprechend distaler Fibulafrakturen.

• Eine zuverlässige Detektion von distalen Fibulafrakturen ist durch das automatisierte Klassifizierungssystem gewährleistet.

• Eine besonders geeignete Methode zur Feature Extraction ist das BOW-Modell.

Key words

ankle - Natural Language Processing - Text Mining - Fibula Fracture - Automatic Classification - Data Set

Publication History

Received: 17 October 2022

Accepted: 18 February 2023

Article published online:
09 May 2023

• References

• 1 Hersh WR, Weiner MG, Embi PJ. et al. Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research. Med Care 2013; 51 (08) S30-S37
• 2 Smith M, Saunders R, Stuckhardt L. et al. Best care at lower cost. National Academies Press; 2014.
• 3 Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci Transl Med 2010; 2 (57) 57cm29
• 4 Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. New England Journal of Medicine 2010; 363 (06) 501-504
• 5 Grundmeier RW, Masino AJ, Casper TC. et al. Identification of long bone fractures in radiology reports using natural language processing to support healthcare quality improvement. Applied clinical informatics 2016; 7 (04) 1051
• 6 Pons E, Braun LM, Hunink MM. et al. Natural language processing in radiology: a systematic review. Radiology 2016; 279 (02) 329-343
• 7 Gerbel S, Laser H, Schönfeld N. et al. The Hannover Medical School Enterprise Clinical Research Data Warehouse: 5 Years of Experience. In: International Conference on Data Integration in the Life Sciences. Springer;. 2018: 182-194
• 8 Hassanpour S, Langlotz CP. Information extraction from multi-institutional radiology reports. Artificial intelligence in medicine 2016; 66: 29-39
• 9 Reddy CK, Aggarwal CC. Healthcare data analytics. Vol. 36. CRC Press; 2015.
• 10 Hearst MA. Untangling text data mining. In: Proceedings of the 37^th Annual meeting of the Association for Computational Linguistics. 1999: 3-10
• 11 Rajkomar A, Oren E, Chen K. et al. Scalable and accurate deep learning with electronic health records. NPJ Digital Medicine 2018; 1 (01) 1-10
• 12 Devlin J, Chang MW, Lee K. et al BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018 [cited 2022 Oct 17]; Available from: https://arxiv.org/abs/1810.04805
• 13 Lee J, Yoon W, Kim S. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Wren J, editor. Bioinformatics. 2019 Sep 10;btz682.
• 14 Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. 2019 [cited 2022 Oct 17]; Available from: https://arxiv.org/abs/1904.05342
• 15 Yamamoto Y, Saito A, Tateishi A. et al. Quantitative diagnosis of breast tumors by morphometric classification of microenvironmental myoepithelial cells using a machine learning approach. Scientific reports 2017; 7 (01) 1-12
• 16 Christodoulou E, Ma J, Collins GS. et al. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of clinical epidemiology 2019; 110: 12-22
• 17 Gougoulias N, Sakellariou A. Ankle Fractures. In: Bentley G. European Surgical Orthopaedics and Traumatology: The EFORT Textbook [Internet]. Berlin, Heidelberg: Springer; 2014: 3735-3765 [cited 2021 Mar 19]. Available from:
• 18 Hasselman CT, Vogt MT, Stone KL. et al. Foot and Ankle Fractures in Elderly White Women: Incidence and Risk Factors. JBJS 2003; 85 (05) 820-824
• 19 Knutsen AR, Sangiorgio SN, Liu C. et al. Distal fibula fracture fixation: Biomechanical evaluation of three different fixation implants. Foot and Ankle Surgery 2016; 22 (04) 278-285
• 20 Neumann MV, Strohm PC, Reising K. et al. Complications after surgical management of distal lower leg fractures. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 2016; 24 (01) 146
• 21 Zuccon G, Wagholikar AS, Nguyen AN. et al. Automatic classification of free-text radiology reports to identify limb fractures using machine learning and the snomed ct ontology. AMIA Summits on Translational Science Proceedings 2013; 2013: 300
• 22 de Bruijn B, Cranney A, O’Donnell S. et al. Identifying wrist fracture patients with high accuracy by automatic categorization of X-ray reports. Journal of the American Medical Informatics Association 2006; 13 (06) 696-698
• 23 Do BH, Wu AS, Maley J. et al. Automatic retrieval of bone fracture knowledge using natural language processing. Journal of digital imaging 2013; 26 (04) 709-713
• 24 Zhixiang X, Chen M, Weinberger K. et al An alternative text representation to TF-IDF and Bag-of-Words [Internet]. arXiv; 2013 [cited 2023 Jan 22]. Available from: http://arxiv.org/abs/1301.6770
• 25 Deisenroth MP, Faisal AA, Ong CS. Dimensionality Reduction and Principal Component Analysis. Math. Mach. Learn. Vol. 80. 2018: 314-344
• 26 Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. the Journal of machine Learning research 2003; 3: 993-1022
• 27 Kim HK, Kim H, Cho S. Bag-of-concepts: Comprehending document representation through clustering words in distributed representation. Neurocomputing 2017; 266: 336-352
• 28 Kim D, Seo D, Cho S. et al. Multi-co-training for document classification using various document representations: TF–IDF, LDA, and Doc2Vec. Information Sciences 2019; 477: 15-29
• 29 Borchert F, Lohr C, Modersohn L. et al GGPONC: A Corpus of German Medical Text with Rich Metadata Based on Clinical Practice Guidelines [Internet]. arXiv; 2020 [cited 2023 Jan 22]. Available from: http://arxiv.org/abs/2007.06400