Radiographic findings in COVID-19: Comparison between AI and radiologist

Abstract

Context: As the burden of COVID-19 enhances, the need of a fast and reliable screening method is imperative. Chest radiographs plays a pivotal role in rapidly triaging the patients. Unfortunately, in low-resource settings, there is a scarcity of trained radiologists. Aim: This study evaluates and compares the performance of an artificial intelligence (AI) system with a radiologist in detecting chest radiograph findings due to COVID-19. Subjects and Methods: The test set consisted of 457 CXR images of patients with suspected COVID-19 pneumonia over a period of three months. The radiographs were evaluated by a radiologist with experience of more than 13 years and by the AI system (NeuraCovid, a web application that pairs with the AI model COVID-NET). Performance of AI system and the radiologist were compared by calculating the sensitivity, specificity and generating a receiver operating characteristic curve. RT-PCR test results were used as the gold standard. Results: The radiologist obtained a sensitivity and specificity of 44.1% and 92.5%, respectively, whereas the AI had a sensitivity and specificity of 41.6% and 60%, respectively. The area under curve for correctly classifying CXR images as COVID-19 pneumonia was 0.48 for the AI system and 0.68 for the radiologist. The radiologist’s prediction was found to be superior to that of the AI with a P VALUE of 0.005. Conclusion: The specificity and sensitivity of detecting lung involvement in COVID-19, by the radiologist, was found to be superior to that by the AI system.

Keywords

Artificial intelligence - chest radiographs - COVID pneumonia - rapid triaging

Publication History

Received: 18 September 2020

Accepted: 15 October 2020

Article published online:
13 July 2021

• References

• 1 Meo SA, Alhowikan AM, Al-Khlaiwi T, Meo IM, Halepoto DM, Iqbal M. et al. Novel coronavirus 2019-nCoV: prevalence, biological and clinical characteristics comparison with SARS-CoV and MERS-CoV. Eur Rev Med Pharmacol Sci 2020; 24: 2012-9
• 2 Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y. et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020; 395: 497-506
• 3 WHO’s Certified [Internet]. Pneumonia of unknown cause–China’, Emergencies preparedness, response. Disease outbreak news. Available from: https://www.who.int/csr/don/05-january-2020-pneumonia-of-unkown-cause-china/en/. [Cited on 2020 Sept 9].
• 4 Worldometers.info[Internet].COVID-19 coronavirus pandemic. Available at: https://www.worldometers.info/coronavirus/. [Updated on 2020 Sept 18; Cited on 2020 Sept 18].
• 5 Wu Z, McGoogan JM. Asymptomatic and pre-symptomatic COVID-19 in China. Infect Dis Poverty 2020; 9: 72
• 6 Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J. et al. Clinical characteristics of 138 hospitalized patients with 2019 novel coronavirus–infected pneumonia in Wuhan, China. JAMA 2020; 323: 1061-9
• 7 Corman VM, Landt O, Kaiser M, Molenkamp R, Meijer A, Chu DK. et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eur Surveill 2020; 25: 2000045
• 8 Tahamtan A, Ardebili A. Real-time RT-PCR in COVID-19 detection: Issues affecting the results. Expert Rev Mol Diagn 2020; 20: 453-4
• 9 Watson J, Whiting PF, Brush JE. Interpreting a covid-19 test result. BMJ 2020; 369: m1808
• 10 Yang W, Sirajuddin A, Zhang X, Liu G, Teng Z, Zhao S. et al. The role of imaging in 2019 novel coronavirus pneumonia (COVID-19). Eur Radiol 2020; 30: 4874-82
• 11 Jacobi A, Chung M, Bernheim A, Eber C. Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review. Clin Imaging 2020; 64: 35-42
• 12 Zu ZY, Jiang MD, Xu PP, Chen W, Ni QQ, Lu GM. et al. Coronavirus disease 2019 (COVID-19): A perspective from China. Radiology 2020; 296: E15-25
• 13 Fang Y, Zhang H, Xie J, Lin M, Ying L, Pang P. et al. Sensitivity of chest CT for COVID-19: Comparison to RT-PCR. Radiology 2020; 296: E115-7
• 14 Rubin GD, Ryerson CJ, Haramati LB, Sverzellati N, Kanne JP, Raoof S. et al. The role of chest imaging in patient management during the COVID-19 pandemic: A multinational consensus statement from the Fleischner Society. Chest 2020; 158: 106-16
• 15 Durrani M, Inam ul Haq UK, Yousaf A. Chest X-rays findings in COVID 19 patients at a University Teaching Hospital-A descriptive study. Pak J Med Sci 2020; 36: S22-6
• 16 Ng MY, Lee EY, Yang J, Yang F, Li X, Wang H. et al. Imaging profile of the COVID-19 infection: Radiologic findings and literature review. Radiology: Cardiothoracic Imaging 2020; 2: e200034
• 17 Antonio GE, Ooi CG, Wong KT, Tsui ELH, Wong JSW, Sy ANL. et al. Radiographic-clinical correlation in severe acute respiratory syndrome: Study of 1373 patients in Hong Kong. Radiology 2005; 237: 1081-90
• 18 Toussie D, Voutsinas N, Finkelstein M, Cedillo MA, Manna S, Maron SZ. et al. Clinical and chest radiography features determine patient outcomes in young and middle age adults with COVID-19. [published online ahead of print, 2020 May 14]. Radiology 2020; 29: E197-206
• 19 Wang L, Wong A, Lin ZA. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest x-ray images. arXiv preprint 2020;arXiv: 2003:09871.
• 20 Murphy K, Smits H, Knoops AJ, Korst MB, Samson T, Scholten ET. et al. COVID-19 on the chest radiograph: A multi-reader evaluation of an ai system. Radiology 2020; 296: E166-72
• 21 Ozturk T, Talo M, Yildirim EA, Baloglu UB, Yildirim O, Acharya UR. Automated detection of COVID-19 cases using deep neural networks with X-ray images. Comput Biol Med 2020; 121: 103792