Performance of AI Models in Guiding Endoscopic Procedures: Insights from Emergency and Elective Scenarios

Aims The objective of this study is to assess whether artificial intelligence is a valuable tool for guiding physicians in performing routine endoscopic procedures as well as those conducted in emergency situations.

Methods We developed 120 clinical cases, based on data from existing patients, 30 for each of the following four topics: 1- Emergency EGD's 2- Emergency ERCP for cholangitis 3-Polypectomy 4-Placement of PEG

Each case was submitted to three artificial intelligence models: Google GEMINI, ChatGPT (GPT-4o mini version), and Microsoft Copilot. As virtual gastroenterology experts, these models were evaluated according to their response in relation with each topic :

• Did they recommend urgent or delayed EGD for each case?

• For cases of cholangitis, did they recommend ERCP within 12 hours, within 48–72 hours, or in a delayed ("cold") setting?

• What management did they propose for a polyp based on its endoscopic description?

• Did they indicate the need for or against the placement of a PEG ?

The AI recommendations were recorded in an Excel file and compared with the evaluations of an expert endoscopist for each topic. Statistical analysis was conducted using Jamovi software (version 2.3).

Results Regarding the 1st topic, moderate agreement was observed among the three AI models for urgent endoscopy recommendations (κ=0.540, p<0.001). The agreement between the gastroenterologist and ChatGPT, as well as Microsoft Copilot, was weak (κ=0.267 and 0.203, respectively). However, substantial agreement was observed with GEMINI (κ=0.733, p<0.001). For the 2nd topic, moderate agreement was noted among the three AI models regarding the timing of ERCP (κ=0.583, p<0.001). Microsoft Copilot demonstrated the best agreement with the gastroenterologist, reaching a substantial level (κ=0.663, p<0.001). Regarding the 3rd topic, the AI models performed well in determining the appropriate procedure for managing a colonic polyp. ChatGPT demonstrated an almost perfect agreement with the gastroenterologist (κ=0.954, p<0.001). Finally, the indications for PEG placement showed poorer results overall. A fair agreement was observed among the three AI models (κ=0.375, p<0.001). ChatGPT provided the best results, achieving substantial agreement with the gastroenterologist (κ=0.796, p<0.001).

Conclusions This study highlights the potential of AI models as decision-support tools in endoscopy, with varying levels of agreement depending on the topic. While GEMINI and ChatGPT demonstrated strong performance in specific areas, inconsistencies across models emphasize the need for further optimization.

 

Scroll to top

Conflicts of Interest

Authors do not have any conflict of interest to disclose.

Publikationsverlauf

Artikel online veröffentlicht:
27. März 2025