  Large language models (LLMs) are fundamentally transforming human-facing
applications in the health and well-being domains: boosting patient engagement,
accelerating clinical decision-making, and facilitating medical education.
Although state-of-the-art LLMs have shown superior performance in several
conversational applications, evaluations within nutrition and diet applications
are still insufficient. In this paper, we propose to employ the Registered
Dietitian (RD) exam to conduct a standard and comprehensive evaluation of
state-of-the-art LLMs, GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, assessing
both accuracy and consistency in nutrition queries. Our evaluation includes
1050 RD exam questions encompassing several nutrition topics and proficiency
levels. In addition, for the first time, we examine the impact of Zero-Shot
(ZS), Chain of Thought (CoT), Chain of Thought with Self Consistency (CoT-SC),
and Retrieval Augmented Prompting (RAP) on both accuracy and consistency of the
responses. Our findings revealed that while these LLMs obtained acceptable
overall performance, their results varied considerably with different prompts
and question domains. GPT-4o with CoT-SC prompting outperformed the other
approaches, whereas Gemini 1.5 Pro with ZS recorded the highest consistency.
For GPT-4o and Claude 3.5, CoT improved the accuracy, and CoT-SC improved both
accuracy and consistency. RAP was particularly effective for GPT-4o to answer
Expert level questions. Consequently, choosing the appropriate LLM and
prompting technique, tailored to the proficiency level and specific domain, can
mitigate errors and potential risks in diet and nutrition chatbots.
