  This study examines the use of Large Language Models (LLMs) for retrieving
factual information, addressing concerns over their propensity to produce
factually incorrect "hallucinated" responses or to altogether decline to even
answer prompt at all. Specifically, it investigates the presence of
gender-based biases in LLMs' responses to factual inquiries. This paper takes a
multi-pronged approach to evaluating GPT models by evaluating fairness across
multiple dimensions of recall, hallucinations and declinations. Our findings
reveal discernible gender disparities in the responses generated by GPT-3.5.
While advancements in GPT-4 have led to improvements in performance, they have
not fully eradicated these gender disparities, notably in instances where
responses are declined. The study further explores the origins of these
disparities by examining the influence of gender associations in prompts and
the homogeneity in the responses.
