Diagnostic performance and agreement of auditors for evaluation of computer-aided optical polyp diagnosis: Prospective study

Abstract

Background and study aims

Guidelines recommend independent auditing of diagnostic performance for clinical implementation of computer-aided optical polyp diagnosis (CADx). This study evaluated diagnostic performance and interobserver agreement of auditors and offered guidance on conducting CADx audits.

Methods

Images and videos of all ≤ 5-mm polyps from a large, prospective study with systematic activation of CADx were audited by three expert endoscopists. Experts performed independent, blinded diagnostic review including documentation of confidence level. The primary outcome was sensitivity of audit by three experts for high-confidence adenomas compared with pathology. Secondary outcomes included number of reviewers for optimal CADx auditing and interobserver agreement.

Results

Four hundred eighty-seven diminutive polyps were audited (510 patients). Sensitivity was 99.4% (95% confidence interval [CI] 96.0–100) using three experts (Strategy A); 88.7% (95% CI 84.1–92.1) using two experts and one referee (Strategy B); 99% (95% CI 96–99.8), 98.8% (95% CI 95.4–99.8), and 99.4% (95% CI 96.3–100) using two-expert combinations (Strategy C); and 98.2% (95% CI 95.1–99.4), 97.3% (95% CI 94.0–98.9), and 88.9% (95% CI 83.6–92.7) for each expert individually (Strategy D). Among 266 pathology-based adenomas, Strategy A evaluated 160 polyps versus 196, 172, and 170 in Strategy C; and 220, 223, and 207 in Strategy D. Strategy B evaluated all 266 adenomas. Overall interobserver agreement was moderate (kappa 0.52), but very high for high-confidence adenomas (kappa 0.89).

Conclusions

Expert audit for evaluating CADx resulted in high sensitivity and interobserver agreement for high-confidence adenomas. Audit by two experts, with a third expert for arbitration, permitted audit of all polyps and effective assessment of CADx within clinical studies.

Keywords

Endoscopy Lower GI Tract - Polyps / adenomas / ... - GI Pathology - Quality and logistical aspects - Quality management

Publication History

Received: 23 April 2025

Accepted: 03 June 2025

Accepted Manuscript online:
10 June 2025

Article published online:
01 July 2025

• References

• 1 Djinbachian R, Rex DK, von Renteln D. Optical polyp diagnosis in the era or artificial intelligence. Am J Gastroenterol 2024; 120: 1268-1274
• 2 Kessler WR, Imperiale TF, Klein RW. et al. A quantitative assessment of the risks and cost savings of forgoing histologic examination of diminutive polyps. Endoscopy 2011; 43: 683-691
• 3 Rex DK, Kahi C, O’Brien M. et al. The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on real-time endoscopic assessment of the histology of diminutive colorectal polyps. Gastrointest Endosc 2011; 73: 419-422
• 4 Ahmad A, Moorghen M, Wilson A. et al. Implementation of optical diagnosis with a "resect and discard" strategy in clinical practice: DISCARD3 study. Gastrointest Endosc 2022; 96: 1021-1032.e2
• 5 Barua I, Wieszczy P, Kudo SE. et al. Real-time artificial intelligence-based optical diagnosis of neoplastic polyps during colonoscopy. NEJM Evid 2022; 1: EVIDoa2200003
• 6 Weigt J, Repici A, Antonelli G. et al. Performance of a new integrated computer-assisted system (CADe/CADx) for detection and characterization of colorectal neoplasia. Endoscopy 2022; 54: 180-184
• 7 Taghiakbari M, Rex DK, Pohl H. et al. Pragmatic resect and discard implementation using computer-assisted optical polyp diagnosis. Gastroenterology 2025; 168: 154-156.e2
• 8 Bisschops R, East JE, Hassan C. et al. Advanced imaging for detection and differentiation of colorectal neoplasia: European Society of Gastrointestinal Endoscopy (ESGE) Guideline - update 2019. Endoscopy 2019; 51: 1155-1179
• 9 Cohen JF, Korevaar DA, Altman DG. et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open 2016; 6: e012799
• 10 Djinbachian R, Haumesser C, Taghiakbari M. et al. Autonomous artificial intelligence vs artificial intelligence-assisted human optical diagnosis of colorectal polyps: a randomized controlled trial. Gastroenterology 2024; 167: 392-399.e2
• 11 WHO Classification of Tumours Editorial Board. Digestive System Tumours, WHO Classification of Tumours. Lyon, France: International Agency for Research on Cancer; 2019
• 12 Houwen BBSL, Hassan C, Coupé VMH. et al. Definition of competence standards for optical diagnosis of diminutive colorectal polyps: European Society of Gastrointestinal Endoscopy (ESGE) Position Statement. Endoscopy 2022; 54: 88-99
• 13 Rex DK, Bhavsar-Burke I, Buckles D. et al. Artificial intelligence for real-time prediction of the histology of colorectal polyps by general endoscopists. Ann Intern Med 2024; 177: 911-918
• 14 Shahidi N, Rex DK, Kaltenbach T. et al. Use of endoscopic impression, artificial intelligence, and pathologist interpretation to resolve discrepancies between endoscopy and pathology analyses of diminutive colorectal polyps. Gastroenterology 2020; 158: 783-785.e1
• 15 Ponugoti P, Rastogi A, Kaltenbach T. et al. Disagreement between high confidence endoscopic adenoma prediction and histopathological diagnosis in colonic lesions ≤ 3 mm in size. Endoscopy 2019; 51: 221-226
• 16 Ahmad A, Saunders BP. Photodocumentation in colonoscopy: the need to do better?. Front Gastroenterol 2021; 13: 337-341
• 17 Rogart JN, Jain D, Siddiqui UD. et al. Narrow-band imaging without high magnification to differentiate polyps during real-time colonoscopy: improvement with experience. Gastrointest Endosc 2008; 68: 1136-1145
• 18 Chen PJ, Lin MC, Lai MJ. et al. Accurate classification of diminutive colorectal polyps using computer-aided analysis. Gastroenterology 2018; 154: 568-575
• 19 Kim DH, Fournier S, Medawar E. et al. Prospective video-based study assessing effect of computer-assisted optical diagnosis on distinguishing serrated, hyperplastic, and adenomatous colorectal polyps. Digest Dis Sci 2025; 70: 1477-1485
• 20 Soons E, Bisseling TM, van der Post RS. et al. The Workgroup Serrated Polyps and Polyposis (WASP) classification for optical diagnosis of colorectal diminutive polyps with iScan and the impact of the revised World Health Organization (WHO) criteria. United European Gastroenterol J 2021; 9: 819-828
• 21 Vennelaganti S, Cuatrecasas M, Vennalaganti P. et al. Interobserver agreement among pathologists in the differentiation of sessile serrated from hyperplastic polyps. Gastroenterology 2021; 160: 452-454.e1
• 22 Yoon H, Martin A, Benamouzig R. et al. Inter-observer agreement on histological diagnosis of colorectal polyps: the APACC study. [Article in French] Gastroenterol Clin Biol 2002; 26: 220-224
• 23 Foss FA, Milkins S, McGregor AH. Inter-observer variability in the histological assessment of colorectal polyps detected through the NHS Bowel Cancer Screening Programme. Histopathology 2012; 61: 47-52
• 24 van Putten PG, Hol L, van Dekken H. et al. Inter-observer variation in the histological diagnosis of polyps in colorectal cancer screening. Histopathology 2011; 58: 974-981
• 25 Ahmad A, Hearing S, Stebbing J. et al. Implementation of optical diagnosis with a resect and discard strategy for diminutive polyps in the English Bowel Cancer Screening Programme. Endoscopy 2025; 57: 115
• 26 Rastogi A, Pondugula K, Bansal A. et al. Recognition of surface mucosal and vascular patterns of colon polyps by using narrow-band imaging: interobserver and intraobserver agreement and prediction of polyp histology. Gastrointest Endosc 2009; 69: 716-722
• 27 Costantini M, Sciallero S, Giannini A. et al. Interobserver agreement in the histologic diagnosis of colorectal polyps. the experience of the multicenter adenoma colorectal study (SMAC). J Clin Epidemiol 2023; 56: 209-214
• 28 Jensen P, Krogsgaard MR, Christiansen J. et al. Observer variability in the assessment of type and dysplasia of colorectal adenomas, analyzed using kappa statistics. Dis Colon Rectum 1995; 38: 195-198