Semi-Automated Annotation tool Outperforms Trained Medical Students and is Comparable to Clinical Expert Performance for Frame-Level Detection of Colorectal Polyps

Aims Training of deep learning systems requires an enormous amount of labeled data. This data must ideally cover the entire range of polyp appearances in real life, but also the whole possible range of image qualities and polyp locations and sizes. Expert labelling of each frame in a polyp video is, therefore, the most robust way for constructing a training set, but this is very time-consuming and currently represents a major barrier for widespread implementation of AI in endoscopy. In this study, two alternative approaches are evaluated, an innovative semi-automated labelling tool and trained medical students providing annotations.

Methods 20 unique polyp white light videos containing 6282 frames (14 adenomas and 6 sessile serrated lesions confirmed by histopathology, mean size 7mm, Olympus) were annotated with bounding boxes by a clinical expert. These annotations are used as the gold standard for comparison. Two cheaper annotation methods were then applied to evaluate their validity and relative performance: (1) a semi-automated labelling technique – this tool only requires 3 manually annotated video frames, from which a representation of the polyp is learned and transferred automatically to all the other frames in the video; (2) independent manual labelling of each video by three medical students – following a training module with polyp images and videos.

Tab. 1

	Sensitivity	PPV	Adjudicated PPV	Time (mins)
Student 1	74,38 ± 27,30	88,52 ± 30,51	89,92 ± 15,34	264
Student 2	63,08 ± 20,27	94,69 ± 22,30	95,00 ± 07,47	1208
Student 3	66,97 ± 27,37	94,77 ± 22,32	95,00 ± 12,30	234
Semi-automated	94,40 ± 06,22	97,17 ± 05,87	98,97 ± 14,04	25

Results The mean and standard deviation of the frame-level sensitivity, positive predictive value (PPV) and adjudicated PPV (for borderline low-quality frames) over all videos are provided in table 1. The semi-automated method significantly outperforms all three students on sensitivity and annotation time (paired t-test, p-value < 0.05), while also achieving the highest value for PPV, both before and after adjudication.

Conclusions A semi-automated labelling tool is a faster, more efficient and valid approach for polyp detection. It outperforms three medical students, specifically trained for polyp recognition and is comparable to clinical expert performance.

Citation: Eelbode T, Ahmad OF, Sinonquel P et al. OP140 SEMI-AUTOMATED ANNOTATION TOOL OUTPERFORMS TRAINED MEDICAL STUDENTS AND IS COMPARABLE TO CLINICAL EXPERT PERFORMANCE FOR FRAME-LEVEL DETECTION OF COLORECTAL POLYPS. Endoscopy 2021; 53: S58.

Publication History

Article published online:
19 March 2021