SUN-697 - Large Language Models, Asians, and the Metabolic Syndrome

Background: AI LLMs have been used by the lay public for health queries. South Asians have a proclivity towards metabolic syndrome and therefore a higher risk for DM, cardiovascular disease, and stroke. 37% of Asian Americans have pre-diabetes. According to the WHO, the target BMI range for Asians (18.5-22.9 ) is narrower than it is for Caucasians (18.5-24.9). This highlights a need for a nuanced understanding of obesity targets for Asian patients in America.

Purpose: Do LLMs provide health education for obese, pre-diabetic Asian patients

Methods: ChatGPT (ChatG), Gemini (GEM), Claude (CLA), Grok, and Meta were posed the following query: “I am an overweight 52-year-old (man/woman) South Asian/East Asian with an HbA1c of 5.7%, prediabetic. What should I do?”

Each LLM was queried only once to avoid additional training bias from repeated queries. We compared the lengths and readability of the responses. Readability was quantified using the Flesch-Kincaid Grade Level (FKGL) analysis. We also analyzed the responses for selected physician-determined keywords. Our chosen keywords were “cardiovascular risk”, “World Health Organization (WHO)”, “GLP-1”, “BMI”, and “Metabolic Syndrome.”

Results:
FKGL range across all LLMs =1.9-11.3; ChatG=9.9-10.9; GEM= 7.9-8.9; CLA=9.3-10.4; Grok=8.6-11.3; Meta=1.9-4.7. Of note, the FKGL of Meta’s responses for South Asian (SA) Male and Female=4.9; East Asian (EA) Male=1.9; EA Female =3.4.
Word counts: ChatG SA Male=502, SA Female=528; EA Male=453, EA Female=437; GEM SA Male=519, SA Female=475, EA Male=608, EA Female=613; CLA SA Male=274, SA Female=272; EA Male=259; EA Female=267; Grok SA Male=592, SA Female=614, EA Male=452; EA Female=521; Meta SA Male= 147, SA Female=147, EA Male=45, EA Female=79. Four out of five LLMs (ChatGPT, Grok, Claude, and Meta) produced longer responses to the South Asian prompts than to the East Asian prompts. However, Gemini produced longer responses for East Asian prompts. None of the five LLMs mentioned the keywords “cardiovascular risk”, “WHO”, “GLP-1”, or “Metabolic Syndrome.” All LLMs, excluding Meta, mentioned the keyword “BMI .” Meta had the shortest responses (45-147 words), the response with the lowest reading level for East Asian males at second grade, and included no key words.

Conclusion: Our analysis revealed that LLMs still require physician oversight and feedback to provide comprehensive, clinically-relevant responses to Asian patients struggling with obesity and prediabetes.

*Unless otherwise noted, all abstracts presented at ENDO must not be released to the press or the public until the date and time of presentation. For oral presentations, the abstracts are embargoed until the session begins. The Endocrine Society reserves the right to lift the embargo on specific abstracts that are selected for promotion prior to or during ENDO.*