  Knights and knaves problems represent a classic genre of logical puzzles
where characters either tell the truth or lie. The objective is to logically
deduce each character's identity based on their statements. The challenge
arises from the truth-telling or lying behavior, which influences the logical
implications of each statement. Solving these puzzles requires not only direct
deductions from individual statements, but the ability to assess the
truthfulness of statements by reasoning through various hypothetical scenarios.
As such, knights and knaves puzzles serve as compelling examples of
suppositional reasoning. In this paper, we introduce $\textit{TruthQuest}$, a
benchmark for suppositional reasoning based on the principles of knights and
knaves puzzles. Our benchmark presents problems of varying complexity,
considering both the number of characters and the types of logical statements
involved. Evaluations on $\textit{TruthQuest}$ show that large language models
like Llama 3 and Mixtral-8x7B exhibit significant difficulties solving these
tasks. A detailed error analysis of the models' output reveals that
lower-performing models exhibit a diverse range of reasoning errors, frequently
failing to grasp the concept of truth and lies. In comparison, more proficient
models primarily struggle with accurately inferring the logical implications of
potentially false statements.
