Present Limitations of Artificial Intelligence in the Emergency Setting – Performance Study of a Commercial, Computer-Aided Detection Algorithm for Pulmonary Embolism

Abstract

Purpose Since artificial intelligence is transitioning from an experimental stage to clinical implementation, the aim of our study was to evaluate the performance of a commercial, computer-aided detection algorithm of computed tomography pulmonary angiograms regarding the presence of pulmonary embolism in the emergency room.

Materials and Methods This retrospective study includes all pulmonary computed tomography angiogram studies performed in a large emergency department over a period of 36 months that were analyzed by two radiologists experienced in emergency radiology to set a reference standard. Original reports and computer-aided detection results were compared regarding the detection of lobar, segmental, and subsegmental pulmonary embolism. All computer-aided detection findings were analyzed concerning the underlying pathology. False-positive findings were correlated to the contrast-to-noise ratio.

Results Expert reading revealed pulmonary embolism in 182 of 1229 patients (49 % men, 10–97 years) with a total of 504 emboli. The computer-aided detection algorithm reported 3331 findings, including 258 (8 %) true-positive findings and 3073 (92 %) false-positive findings. Computer-aided detection analysis showed a sensitivity of 47 % (95 %CI: 33–61 %) on the lobar level and 50 % (95 %CI 43–56 %) on the subsegmental level. On average, there were 2.25 false-positive findings per study (median 2, range 0–25). There was no significant correlation between the number of false-positive findings and the contrast-to-noise ratio (Spearman’s Rank Correlation Coefficient = 0.09). Soft tissue (61.0 %) and pulmonary veins (24.1 %) were the most common underlying reasons for false-positive findings.

Conclusion Applied to a population at a large emergency room, the tested commercial computer-aided detection algorithm faced relevant performance challenges that need to be addressed in future development projects.

Key Points: 

• Computed tomography pulmonary angiograms are frequently acquired in emergency radiology.

• Computer-aided detection algorithms (CADs) can support image analysis.

• CADs face challenges regarding false-positive and false-negative findings.

• Radiologists using CADs need to be aware of these limitations.

• Further software improvements are necessary ahead of implementation in the daily routine.

Citation Format

• Müller-Peltzer K, Kretzschmar L, Negrão de Figueiredo G et al. Present Limitations of Artificial Intelligence in the Emergency Setting – Performance Study of a Commercial, Computer-Aided Detection Algorithm for Pulmonary Embolism. Fortschr Röntgenstr 2021; 193: 1436 – 1444

Key words

pulmonary embolism - CAD - CT pulmonary angiogram - computer-aided detection

Publication History

Received: 31 December 2020

Accepted: 04 May 2021

Article published online:
05 August 2021

• References

• 1 Calder KK, Herbert M, Henderson SO. The mortality of untreated pulmonary embolism in emergency department patients. Ann Emerg Med 2005; 45: 302-310 DOI: 10.1016/j.annemergmed.2004.10.001.
• 2 Wendelboe AM, Raskob GE. Global Burden of Thrombosis: Epidemiologic Aspects. Circ Res 2016; 118: 1340-1347 DOI: 10.1161/circresaha.115.306841.
• 3 Diehm C, Noppeney T, Nüllen H. Epidemiologie der venösen Thromboembolie. Gefässchirurgie 2012; 17: 275-279 DOI: 10.1007/s00772-011-0929-8.
• 4 Sharma S, Lucas CD. Increasing use of CTPA for the investigation of suspected pulmonary embolism. Postgrad Med 2017; 129: 193-197 DOI: 10.1080/00325481.2017.1281084.
• 5 Schissler A, Rozenshtein A, Kulon M. et al. CT Pulmonary Angiography: Increasingly Diagnosing Less Severe Pulmonary Emboli. PloS one 2013; 8: e65669 DOI: 10.1371/journal.pone.0065669.
• 6 Kligerman SJ, Lahiji K, Galvin JR. et al. Missed pulmonary emboli on CT angiography: assessment with pulmonary embolism-computer-aided detection. Am J Roentgenol 2014; 202: 65-73 DOI: 10.2214/ajr.13.11049.
• 7 Hawley PC, Hawley MP. Difficulties in diagnosing pulmonary embolism in the obese patient: a literature review. Vasc Med 2011; 16: 444-451 DOI: 10.1177/1358863x11422571.
• 8 Buhmann S, Herzog P, Liang J. et al. Clinical evaluation of a computer-aided diagnosis (CAD) prototype for the detection of pulmonary embolism. Acad Radiol 2007; 14: 651-658 DOI: 10.1016/j.acra.2007.02.007.
• 9 Moore AJE, Wachsmann J, Chamarthy MR. et al. Imaging of acute pulmonary embolism: an update. Cardiovasc Diagn Ther 2018; 8: 225-243 DOI: 10.21037/cdt.2017.12.01.
• 10 Eyer BA, Goodman LR, Washington L. Clinicians' response to radiologists' reports of isolated subsegmental pulmonary embolism or inconclusive interpretation of pulmonary embolism using MDCT. Am J Roentgenol 2005; 184: 623-628 DOI: 10.2214/ajr.184.2.01840623.
• 11 Wiener RS, Schwartz LM, Woloshin S. Time trends in pulmonary embolism in the United States: evidence of overdiagnosis. Arch Intern Med 2011; 171: 831-837 DOI: 10.1001/archinternmed.2011.178.
• 12 den Exter PL, van Es J, Klok FA. et al Risk profile and clinical outcome of symptomatic subsegmental acute pulmonary embolism. Blood 2013; 122: 1144-1149 ; quiz 1329. DOI: 10.1182/blood-2013-04-497545.
• 13 Konstantinides SV, Meyer G, Becattini C. et al. 2019 ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS): The Task Force for the diagnosis and management of acute pulmonary embolism of the European Society of Cardiology (ESC). European Heart Journal 2019; 41: 543-603 DOI: 10.1093/eurheartj/ehz405.
• 14 Tajbakhsh N, Gotway MB, Liang J. Computer-Aided Pulmonary Embolism Detection Using a Novel Vessel-Aligned Multi-planar Image Representation and Convolutional Neural Networks. Cham: Springer International Publishing; 2015: 62-69
• 15 Lahiji K, Kligerman S, Jeudy J. et al. Improved accuracy of pulmonary embolism computer-aided detection using iterative reconstruction compared with filtered back projection. Am J Roentgenol 2014; 203: 763-771 DOI: 10.2214/ajr.13.11838.
• 16 Wittenberg R, Peters JF, Sonnemans JJ. et al. Computer-assisted detection of pulmonary embolism: evaluation of pulmonary CT angiograms performed in an on-call setting. Eur Radiol 2010; 20: 801-806 DOI: 10.1007/s00330-009-1628-7.
• 17 Wittenberg R, Peters JF, Weber M. et al. Stand-alone performance of a computer-assisted detection prototype for detection of acute pulmonary embolism: a multi-institutional comparison. Br J Radiol 2012; 85: 758-764 DOI: 10.1259/bjr/26769569.
• 18 Özkan H, Osman O, Şahin S. et al. A novel method for pulmonary embolism detection in CTA images. Comput Methods Programs Biomed 2014; 113: 757-766 DOI: 10.1016/j.cmpb.2013.12.014.
• 19 Krissak R, Henzler T, Reichert M. et al. Enhanced visualization of lung vessels for diagnosis of pulmonary embolism using dual energy CT angiography. Invest Radiol 2010; 45: 341-346 DOI: 10.1097/RLI.0b013e3181dfda37.
• 20 Wittenberg R, Peters JF, Sonnemans JJ. et al. Impact of Image Quality on the Performance of Computer-Aided Detection of Pulmonary Embolism. American Journal of Roentgenology 2011; 196: 95-101 DOI: 10.2214/Am J Roentgenol.09.4165.
• 21 Masoudi M, Pourreza HR, Saadatmand-Tarzjan M. et al. A new dataset of computed-tomography angiography images for computer-aided detection of pulmonary embolism. Sci Data 2018; 5: 180180 DOI: 10.1038/sdata.2018.180.
• 22 Ma G, Dou Y, Dang S. et al. Influence of Monoenergetic Images at Different Energy Levels in Dual-Energy Spectral CT on the Accuracy of Computer-Aided Detection for Pulmonary Embolism. Acad Radiol 2019; 26: 967-973 DOI: 10.1016/j.acra.2018.09.007.
• 23 Kröger JR, Hickethier T, Pahn G. et al. Influence of spectral detector CT based monoenergetic images on the computer-aided detection of pulmonary artery embolism. Eur J Radiol 2017; 95: 242-248 DOI: 10.1016/j.ejrad.2017.08.034.
• 24 Tajbakhsh N, Shin JY, Gotway MB. et al. Computer-aided detection and visualization of pulmonary embolism using a novel, compact, and discriminative image representation. Med Image Anal 2019; 58: 101541 DOI: 10.1016/j.media.2019.101541.
• 25 Zhang C, Sun M, Wei Y. et al. Automatic segmentation of arterial tree from 3D computed tomographic pulmonary angiography (CTPA) scans. Comput Assist Surg (Abingdon) 2019; 24: 79-86 DOI: 10.1080/24699322.2019.1649077.