  Advancements in deep learning have generated a large-scale interest in the
development of foundational deep learning models. The development of Large
Language Models (LLM) has evolved as a transformative paradigm in
conversational tasks, which has led to its integration and extension even in
the critical domain of healthcare. With LLMs becoming widely popular and their
public access through open-source models and integration with other
applications, there is a need to investigate their potential and limitations.
One such crucial task where LLMs are applied but require a deeper understanding
is that of self-diagnosis of medical conditions based on bias-validating
symptoms in the interest of public health. The widespread integration of Gemini
with Google search and GPT-4.0 with Bing search has led to a shift in the trend
of self-diagnosis using search engines to conversational LLM models. Owing to
the critical nature of the task, it is prudent to investigate and understand
the potential and limitations of public LLMs in the task of self-diagnosis. In
this study, we prepare a prompt engineered dataset of 10000 samples and test
the performance on the general task of self-diagnosis. We compared the
performance of both the state-of-the-art GPT-4.0 and the fee Gemini model on
the task of self-diagnosis and recorded contrasting accuracies of 63.07% and
6.01%, respectively. We also discuss the challenges, limitations, and potential
of both Gemini and GPT-4.0 for the task of self-diagnosis to facilitate future
research and towards the broader impact of general public knowledge.
Furthermore, we demonstrate the potential and improvement in performance for
the task of self-diagnosis using Retrieval Augmented Generation.
