  Large language models (LLMs) are widely used in decision-making, but their
reliability, especially in critical tasks like healthcare, is not
well-established. Therefore, understanding how LLMs reason and make decisions
is crucial for their safe deployment. This paper investigates how the
uncertainty of responses generated by LLMs relates to the information provided
in the input prompt. Leveraging the insight that LLMs learn to infer latent
concepts during pretraining, we propose a prompt-response concept model that
explains how LLMs generate responses and helps understand the relationship
between prompts and response uncertainty. We show that the uncertainty
decreases as the prompt's informativeness increases, similar to epistemic
uncertainty. Our detailed experimental results on real datasets validate our
proposed model.
