Assessing AI for Clinical Decision-Making in the Management of Acute Cholangitis

Aims To evaluate the performance of Google's large language model (Gemini) in adhering to Tokyo Guidelines 2018 (TG18) for acute cholangitis management. We assessed the model's accuracy in disease severity classification, treatment recommendations, and its performance across different severity grades using real-world clinical data.

Methods This observational cross-sectional study analyzed 743 patients from the MIMIC-III dataset (2001-2012) diagnosed with acute cholangitis. Gemini was provided standardized clinical summaries and asked to classify disease severity and recommend management according to TG18. Performance was evaluated using a 4-point scoring system assessing severity grading, antibiotic recommendations, biliary drainage timing, and ICU admission decisions. Statistical analysis included McNemar's test and Cohen's kappa for agreement assessment.

Results Gemini achieved 92.1% accuracy (684/743 cases) in severity classification, with a concordance coefficient of 0.88 (95% CI: 0.85-0.91). Performance varied by severity: Grade I (95.7%), Grade II (94.2%), and Grade III (87.9%). The model demonstrated high accuracy in antibiotic selection (95.6% of cases) and ICU admission recommendations for Grade III patients (90.2%). Biliary drainage timing recommendations showed 91.3% overall accuracy. The median guideline adherence score was 3.4 (IQR: 3-4) on a 4-point scale.

Conclusions Gemini demonstrated high accuracy in following TG18 recommendations for acute cholangitis management, particularly in severity classification and antibiotic selection. While performance was strong across all severity grades, there was slightly lower accuracy in managing Grade III cases. These findings suggest potential utility for AI in supporting clinical decision-making for acute cholangitis, though maintaining clinical oversight remains crucial. Future prospective studies should evaluate the impact of AI integration on patient outcomes.

Publication History

Article published online:
27 March 2025