SAT-537 - Can Mds Create A LLM-Chatbot for Diabetes Patient Education?

Background: Patients with diabetes face significant health literacy barriers. Educational materials and general-purpose large language models (LLMs) often exceed recommended reading levels. LLMs have been shown to generate incomplete or inaccurate medical information. Clinicians currently have limited tools to control the accuracy, safety, and emphasis of AI-generated pt education.

Purpose: Can MDs develop a no-cost chatbot for diabetes education?

Methods: We created a web-based chatbot using a retrieval-augmented generation (RAG) architecture. Educational content was sourced from patient education materials on the American Diabetes Association website (https://diabetes.org/). Article texts were segmented into chunks, embedded using the Gemini text-embedding-004 model, and stored in a local vector database. User queries were answered using Gemini 2.5 Flash. Responses were limited to (1) only use information provided from the articles, (2) respond at a 6th-grade reading level, (3) avoid diagnosing or giving personal advice, and (4) emphasize diabetes-related eye examinations, kidney monitoring, foot evaluation, and cardiovascular risk. We compared the chatbot against ChatGPT-5 mini, Claude.ai Opus 4.5, and Gemini 3 Fast in their responses to this query: “I was just diagnosed with diabetes. I am an adult. What should I do?” All LLMs were prompted to respond at a 6th grade reading level. Responses were analyzed for word count and Flesch Readability Ease score (FR: 80-90=6th grade; 70-80=7th grade).

Results:
Word counts, FR: Chatbot (142 words, FR: 80.8) vs. ChatGPT (248, 79.1) vs Claude.ai (307, 79.3) vs Gemini (404, 86.6). The physician-developed chatbot response had the fewest words with an avg reading level of 6th grade, while ChatGPT and Claude generated responses at a 7th grade level. While all models recommended working with a doctor, only the physician-developed chatbot consistently emphasized the effect of diabetes on the eye, kidney, foot, and heart. ChatGPT only mentioned getting eyes and kidneys checked. Claude.ai did not mention any of these risk factors. Gemini only mentioned foot monitoring, and did not mention the others.

Conclusion: A physician-developed, retrieval-based chatbot can be a scalable, low-cost strategy to enhance health literacy and patient education in endocrinology.

*Unless otherwise noted, all abstracts presented at ENDO must not be released to the press or the public until the date and time of presentation. For oral presentations, the abstracts are embargoed until the session begins. The Endocrine Society reserves the right to lift the embargo on specific abstracts that are selected for promotion prior to or during ENDO.*