A deep learning-based system for real-time image reporting during esophagogastroduodenoscopy: a multicenter study

Abstract

Background and study aims Endoscopic reports are essential for the diagnosis and follow-up of gastrointestinal diseases. This study aimed to construct an intelligent system for automatic photo documentation during esophagogastroduodenoscopy (EGD) and test its utility in clinical practice.

Patients and methods Seven convolutional neural networks trained and tested using 210,198 images were integrated to construct the endoscopic automatic image reporting system (EAIRS). We tested its performance through man-machine comparison at three levels: internal, external, and prospective test. Between May 2021 and June 2021, patients undergoing EGD at Renmin Hospital of Wuhan University were recruited. The primary outcomes were accuracy for capturing anatomical landmarks, completeness for capturing anatomical landmarks, and detected lesions.

Results The EAIRS outperformed endoscopists in retrospective internal and external test. A total of 161 consecutive patients were enrolled in the prospective test. The EAIRS achieved an accuracy of 95.2% in capturing anatomical landmarks in the prospective test. It also achieved higher completeness on capturing anatomical landmarks compared with endoscopists: (93.1% vs. 88.8%), and was comparable to endoscopists on capturing detected lesions: (99.0% vs. 98.0%).

Conclusions The EAIRS can generate qualified image reports and could be a powerful tool for generating endoscopic reports in clinical practice.

Publication History

Received: 20 July 2021

Accepted after revision: 03 December 2021

Article published online:
10 March 2022

• References

• 1 Rey J-F, El Hajjar A, Gholami R. Photography in digestive endoscopy. In: Pasquali P. Photography in Clinical Medicine. Cham, Switzerland: Springer; 2020: 415-428
• 2 Rey J-F, Lambert R. ESGE Quality Assurance Committee. ESGE recommendations for quality control in gastrointestinal endoscopy: guidelines for image documentation in upper and lower GI endoscopy. Endoscopy 2001; 33: 901-903 DOI: 10.1055/s-2001-42537. (PMID: 11605605)
• 3 Bisschops R, Areia M, Coron E. et al. Performance measures for upper gastrointestinal endoscopy: a European Society of Gastrointestinal Endoscopy (ESGE) quality improvement initiative. Endoscopy 2016; 48: 843-864 DOI: 10.1055/s-0042-113128. (PMID: 27548885)
• 4 ASGE/ACG Task Force on Quality in Endoscopy. Quality indicators for GI endoscopic procedures. Gastrointest Endosc 2015; 81: 1-80
• 5 Chiu PWY, Uedo N, Singh R. et al. An Asian consensus on standards of diagnostic upper endoscopy for neoplasia. Gut 2019; 68: 186-197 DOI: 10.1136/gutjnl-2018-317111. (PMID: 30420400)
• 6 Bretthauer M, Aabakken L, Dekker E. et al. Requirements and standards facilitating quality improvement for reporting systems in gastrointestinal endoscopy: European Society of Gastrointestinal Endoscopy (ESGE) Position Statement. Endoscopy 2016; 48: 291-294 DOI: 10.1055/s-0042-100186. (PMID: 26841269)
• 7 Beg S, Ragunath K, Wyman A. et al. Quality standards in upper gastrointestinal endoscopy: a position statement of the British Society of Gastroenterology (BSG) and Association of Upper Gastrointestinal Surgeons of Great Britain and Ireland (AUGIS). Gut 2017; 66: 1886-1899 DOI: 10.1136/gutjnl-2017-314109. (PMID: 28821598)
• 8 Rutter MD, Rees CJ. Quality in gastrointestinal endoscopy. Endoscopy 2014; 46: 526-528 DOI: 10.1055/a-0889-9586. (PMID: 31075800)
• 9 Miller AT, Sedlack RE, Coyle WJ. et al. Competency in esophagogastroduodenoscopy: a validated tool for assessment and generalizable benchmarks for gastroenterology fellows. Gastrointest Endosc 2019; 90: 613-620.e611 DOI: 10.1016/j.gie.2019.05.024. (PMID: 31121154)
• 10 Lee HL, Eun CS, Lee OY. et al. When do we miss synchronous gastric neoplasms with endoscopy?. Gastrointest Endosc 2010; 71: 1159-1165 DOI: 10.1016/j.gie.2010.01.011. (PMID: 20381041)
• 11 Robertson DJ, Lawrence LB, Shaheen NJ. et al. Quality of colonoscopy reporting: a process of care study. Am J Gastroenterol 2002; 97: 2651-2656 DOI: 10.1111/j.1572-0241.2002.06044.x. (PMID: 12385455)
• 12 Conway JD, Adler DG, Diehl DL. et al. Endoscopic electronic medical record systems. Gastrointest Endosc 2008; 67: 590-594 DOI: 10.1016/j.gie.2008.01.040. (PMID: 18374022)
• 13 Litjens G, Kooi T, Bejnordi BE. et al. A survey on deep learning in medical image analysis. Med Image Anal 2017; 42: 60-88 DOI: 10.1016/j.media.2017.07.005. (PMID: 28778026)
• 14 Takeda K, Kudo S-E, Mori Y. et al. Accuracy of diagnosing invasive colorectal cancer using computer-aided endocytoscopy. Endoscopy 2017; 49: 798-802 DOI: 10.1055/s-0043-105486. (PMID: 28472832)
• 15 Kanesaka T, Lee T-C, Uedo N. et al. Computer-aided diagnosis for identifying and delineating early gastric cancers in magnifying narrow-band imaging. Gastrointest Endosc 2018; 87: 1339-1344 DOI: 10.1016/j.gie.2017.11.029. (PMID: 29225083)
• 16 Wu L, Zhang J, Zhou W. et al. Randomised controlled trial of WISENSE, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy. Gut 2019; 68: 2161-2169 DOI: 10.1136/gutjnl-2018-317366. (PMID: 30858305)
• 17 Wu L, Zhou W, Wan X. et al. A deep neural network improves endoscopic detection of early gastric cancer without blind spots. Endoscopy 2019; 51: 522-531 DOI: 10.1055/a-0855-3532. (PMID: 30861533)
• 18 Wu L, He X, Liu M. et al. Evaluation of the effects of an artificial intelligence system on endoscopy quality and preliminary testing of its performance in detecting early gastric cancer: a randomized controlled trial. Endoscopy 2021; DOI: 10.1055/a-1350-5583.
• 19 Wu L, Shang R, Zhou W. et al. Effect of a deep learning-based system on the miss rate of gastric neoplasms during upper gastrointestinal endoscopy: a single-center, tandem, randomized controlled trial. Lancet Gastroenterol Hepatol 2021; DOI: 10.1016/S2468-1253(21)00216-8.
• 20 Gao F, Yoon H, Wu T. et al. A feature transfer enabled multi-task deep learning model on medical imaging. Expert Syst Appl 2020; 143: 112957