Direct comparison of multiple computer-aided polyp detection systems

Abstract

Background and study aims Artificial intelligence (AI)-based systems for computer-aided detection (CADe) of polyps receive regular updates and occasionally offer customizable detection thresholds, both of which impact their performance, but little is known about these effects. This study aimed to compare the performance of different CADe systems on the same benchmark dataset.

Methods 101 colonoscopy videos were used as benchmark. Each video frame with a visible polyp was manually annotated with bounding boxes, resulting in 129 705 polyp images. The videos were then analyzed by three different CADe systems, representing five conditions: two versions of GI Genius, Endo-AID with detection Types A and B, and EndoMind, a freely available system. Evaluation included an analysis of sensitivity and false-positive rate, among other metrics.

Results Endo-AID detection Type A, the earlier version of GI Genius, and EndoMind detected all 93 polyps. Both the later version of GI Genius and Endo-AID Type B missed 1 polyp. The mean per-frame sensitivities were 50.63 % and 67.85 %, respectively, for the earlier and later versions of GI Genius, 65.60 % and 52.95 %, respectively, for Endo-AID Types A and B, and 60.22 % for EndoMind.

Conclusions This study compares the performance of different CADe systems, different updates, and different configuration modes. This might help clinicians to select the most appropriate system for their specific needs.

• References

• 1 Heresbach D, Barrioz T, Lapalus MG. et al. Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies. Endoscopy 2008; 40: 284-290
• 2 Zhao S, Wang S, Pan P. et al. Magnitude, risk factors, and factors associated with adenoma miss rate of tandem colonoscopy: A systematic review and meta-analysis. Gastroenterology 2019; 156: 1661-1674 e1611
• 3 Kamitani Y, Nonaka K, Isomoto H. Current status and future perspectives of artificial intelligence in colonoscopy. J Clin Med 2022; 11: 2923-2937
• 4 Wang P, Berzin TM, Glissen Brown JR. et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut 2019; 68: 1813-1819
• 5 Wang P, Liu P, Glissen Brown JR. et al. Lower adenoma miss rate of computer-aided detection-assisted colonoscopy vs routine white-light colonoscopy in a prospective tandem study. Gastroenterology 2020; 159: 1252-1261 e1255
• 6 Wang P, Liu X, Berzin TM. et al. Effect of a deep-learning computer-aided detection system on adenoma detection during colonoscopy (CADe-DB trial): a double-blind randomised study. Lancet Gastroenterol Hepatol 2020; 5: 343-351
• 7 Liu P, Wang P, Glissen Brown JR. et al. The single-monitor trial: an embedded CADe system increased adenoma detection during colonoscopy: a prospective randomized study. Therap Adv Gastroenterol 2020; 13
• 8 Su JR, Li Z, Shao XJ. et al. Impact of a real-time automatic quality control system on colorectal polyp and adenoma detection: a prospective randomized controlled study (with videos). Gastrointest Endosc 2020; 91: 415-424 e414
• 9 Repici A, Badalamenti M, Maselli R. et al. Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology 2020; 159: 512-520 e517
• 10 Gong D, Wu L, Zhang J. et al. Detection of colorectal adenomas with a real-time computer-aided system (ENDOANGEL): a randomised controlled study. Lancet Gastroenterol Hepatol 2020; 5: 352-61
• 11 Yao L, Zhang L, Liu J. et al. Effect of an artificial intelligence-based quality improvement system on efficacy of a computer-aided detection system in colonoscopy: a four-group parallel study. Endoscopy 2022; 54: 757-768
• 12 Shaukat A, Lichtenstein DR, Somers SC. et al. Computer-aided detection improves adenomas per colonoscopy for screening and surveillance colonoscopy: A randomized trial. Gastroenterology 2022; 163: 732-741
• 13 Gimeno-Garcia AZ, Negrin DH, Hernandez A. et al. Usefulness of a novel computer-aided detection system for colorectal neoplasia: A randomized controlled trial. Gastrointest Endosc 2022; 97: 528-536
• 14 Brand M, Troya J, Krenzer A. et al. Frame-by-frame analysis of a commercially available artificial intelligence polyp detection system in full-length colonoscopies. Digestion 2022; 103: 378-385
• 15 Krenzer A, Makowski K, Hekalo A. et al. Semi-automated machine learning video annotation for gastroenterologists. Stud Health Technol Inform 2021; 281: 484-485
• 16 S3-Leitlinie Kolorektales Karzinom. In: Leitlinienprogramm Onkologie: Deutsche Krebsgesellschaft, Deutsche Krebshilfe, AWMF; 2019. https://www.leitlinienprogramm-onkologie.de/fileadmin/user_upload/Downloads/Leitlinien/Kolorektales_Karzinom/Version_2/LL_KRK_Langversion_2.1.pdf
• 17 Gupta S, Lieberman D, Anderson JC. et al. Recommendations for follow-up after colonoscopy and polypectomy: A consensus update by the U.S. Multi-Society Task Force on Colorectal Cancer. . Gastrointest Endosc 2020; 91: 463-485 e465
• 18 Repici A, Spadaccini M, Antonelli G. et al. Artificial intelligence and colonoscopy experience: lessons from two randomised trials. Gut 2022; 71: 757-765
• 19 Yamada A, Niikura R, Otani K. et al. Automatic detection of colorectal neoplasia in wireless colon capsule endoscopic images using a deep convolutional neural network. Endoscopy 2021; 53: 832-836
• 20 Weigt J, Repici A, Antonelli G. et al. Performance of a new integrated computer-assisted system (CADe/CADx) for detection and characterization of colorectal neoplasia. Endoscopy 2021; 54: 180-184
• 21 Brand M, Troya J, Krenzer A. et al. Development and evaluation of a deep learning model to improve the usability of polyp detection systems during interventions. United Eur Gastroenterol J 2022; 10: 477-484
• 22 Schauer C, Chieng M, Wang M. et al. Artificial intelligence improves adenoma detection rate during colonoscopy. N Z Med J 2022; 135: 22-30
• 23 Fitting D, Krenzer A, Troya J. et al. A video based benchmark data set (ENDOTEST) to evaluate computer-aided polyp detection systems. Scand J Gastroenterol 2022; 57: 1397-1140