AI vs Experts: Concordance in the Evaluation of Lymph node by EUS

Aims The characterisation of lymph nodes via endoscopic ultrasound (EUS) remains challenging for experts, with morphological criteria for malignancy exhibiting low sensitivity and specificity. Inter-observer agreement varies across characteristics such as shape, echogenicity, homogeneity, and size. Artificial intelligence (AI) may enhance diagnostic accuracy and serve as a valuable tool in this context. Our objective was to evaluate the concordance among three expert endosonographers in assessing lymph node characteristics with EUS and predicting malignancy, compared to an AI model, ChatGPT-4.0 (OpenAI, San Francisco, USA).

Methods Three expert endosonographers, blinded to the final diagnosis, assessed 40 videos of lymph nodes. In parallel, the ChatGPT-4.0 model analysed 40 still images, focusing on a defined region of interest. Concordance was measured using the kappa coefficient for each morphological feature (shape, borders, echogenicity, and cortico-medullary differentiation) and diagnostic accuracy in predicting malignancy, compared with the histological diagnosis as the gold standard.

Results The study included 21 benign and 19 malignant lymph nodes. Concordance among endoscopists was moderate for shape (k=0.51), borders (k=0.45), and malignancy prediction (k=0.63), but low for echogenicity (k=0.21) and cortico-medullary differentiation (k=0.26). Diagnostic accuracy varied among experts: Expert 1 achieved an accuracy of 0.80 (95% confidence interval [CI]: 0.67-0.93), Expert 2 an accuracy of 0.68 (95% CI: 0.53-0.83), and Expert 3 an accuracy of 0.80 (95% CI: 0.68-0.93). In comparison, ChatGPT achieved a diagnostic accuracy of 0.60 (95% CI: 0.45-0.66). The overall p-value for comparing diagnostic accuracy was 0.14, indicating no significant difference between the experts and the AI model.

Conclusions Concordance among endoscopists for lymph node characterisation using EUS is suboptimal. The AI model did not demonstrate an improvement in diagnostic accuracy over the experts. The development of specifically trained AI models, such as convolutional neural networks, may hold promise as a diagnostic tool for EUS-based lymph node characterisation.

Publikationsverlauf

Artikel online veröffentlicht:
27. März 2025