SAT-678 - Comparative Accuracy of Large Language Models in Providing Clinical Information on Anti-Obesity Medications

Background: The surge in clinical prescriptions for anti-obesity medications (AOMs), including GLP-1 receptor agonists, has led patients to increasingly utilize generative artificial intelligence (AI) for medical guidance. However, the potential for these models to provide inaccurate clinical data poses a significant risk to patient safety.

Aim: This study evaluated the accuracy of four prominent generative AI platforms in delivering drug-specific information for AOMs, using official package inserts and clinical practice guidelines as the gold standard for verification.
Materials and

Methods: A total of 120 standardized clinical prompts (40 per medication: Saxenda, Wegovy, and Mounjaro) were evaluated across ChatGPT, Gemini, Meta AI, and Copilot. A total of 480 responses were independently audited by two clinical pharmacists and categorized as Correct, Partially Correct, or Incorrect.

Results: ChatGPT achieved the highest accuracy rate (40.0%), followed by Microsoft Copilot (36.7%), Google Gemini (31.7%), and Meta AI (27.5%). Statistically, Meta AI’s performance was significantly inferior to the reference model (p=0.006*), while Copilot (p=0.421) and Gemini (p=0.082) showed no significant variance. Notably, a substantial percentage of responses across all platforms contained partial or incorrect data regarding clinical scenarios.

Conclusion: While generative AI shows potential as a supplementary tool, it currently lacks the necessary reliability for independent clinical use in obesity management. The high incidence of inaccurate responses necessitates that healthcare providers and patients continue to rely strictly on validated clinical guidelines and manufacturer documentation to ensure safe medication practices.

*Unless otherwise noted, all abstracts presented at ENDO must not be released to the press or the public until the date and time of presentation. For oral presentations, the abstracts are embargoed until the session begins. The Endocrine Society reserves the right to lift the embargo on specific abstracts that are selected for promotion prior to or during ENDO.*