SAT-836 - Clinical Uncertainty in Insulin Pump Selection: Comparing Endocrinologist Judgment with AI Recommendations using Chart-Only Data

Background
Insulin pump selection is a complex, individualized decision that needs more information than clinical data such as patient engagement, lifestyle, and technical aptitude. While generative AI systems may be useful in clinical decision support, their ability to replicate clinician pump selection using chart data alone remains unclear. We hypothesized that insulin pump selection based on chart data alone would result in low confidence and limited agreement for both clinicians and a generative AI model, due to the absence of patient-specific contextual information.
Methods
We conducted a pilot study using 22 diabetes patient cases. Four endocrinologists independently reviewed standardized chart summaries (medical history, medications, and laboratory data) and recommended an insulin pump or continuation of injections. Clinicians rated decision difficulty, confidence, and noted additional information that would help improve recommendations. Additionally, we gave the same chart summaries to a generative AI model (GPT-4) to recommend pump recommendations. Agreement was defined as the same answer between the AI recommendation and at least one clinician’s choice.
Results
AI pump recommendations had the same answer with at least one endocrinologist in 12 of 22 cases (54.5%). In the remaining cases, AI recommendations were different from all clinician selections, including four cases in which the AI recommended no pump while most clinicians favored pumps. Additionally, clinicians recommended a broader range of insulin pumps, recommending established and newer pumps, whereas the AI recommended a narrower range, not including newer or less-publicized systems. Clinicians reported moderate decision difficulty (~5.3/10), moderate confidence (~5/10), and additional patient-specific information, such as daily routines, diets, and technical ability, to improve decision-making. The use of chart-only data for pump selection was rated as moderate (~5/10).
Conclusion
Both clinicians and the generative AI model demonstrated limitations in insulin pump selection when patient behavioral information was absent. The AI’s restricted device recommendations and divergence from clinician judgment highlight gaps due to incomplete inputs rather than reasoning capacity alone. Clinician feedback suggests that pump selection relies on using patient aspects not captured in routine medical records. These findings give us information that needs to be included in the next-generation AI decision support tools: patient-reported data, clinician-informed prompt logic, and up-to-date device knowledge. Future work will focus on developing patient-driven AI systems to make a more informed and accurate pump selection.

*Unless otherwise noted, all abstracts presented at ENDO must not be released to the press or the public until the date and time of presentation. For oral presentations, the abstracts are embargoed until the session begins. The Endocrine Society reserves the right to lift the embargo on specific abstracts that are selected for promotion prior to or during ENDO.*