  Large language models (LLMs) have been shown to possess impressive
capabilities, while also raising crucial concerns about the faithfulness of
their responses. A primary issue arising in this context is the management of
(un)answerable queries by LLMs, which often results in hallucinatory behavior
due to overconfidence. In this paper, we explore the behavior of LLMs when
presented with (un)answerable queries. We ask: do models represent the fact
that the question is (un)answerable when generating a hallucinatory answer? Our
results show strong indications that such models encode the answerability of an
input query, with the representation of the first decoded token often being a
strong indicator. These findings shed new light on the spatial organization
within the latent representations of LLMs, unveiling previously unexplored
facets of these models. Moreover, they pave the way for the development of
improved decoding techniques with better adherence to factual generation,
particularly in scenarios where query (un)answerability is a concern.
