  Recent empirical results have sparked a debate about whether or not Large
Language Models (LLMs) are capable of Theory of Mind (ToM). While some have
found LLMs to be successful on ToM evaluations such as the False Belief task
(Kosinski, 2023), others have argued that LLMs solve these tasks by exploiting
spurious correlations -- not representing beliefs -- since they fail on trivial
alterations to these tasks (Ullman, 2023). In this paper, we introduce SCALPEL:
a technique to generate targeted modifications for False Belief tasks to test
different specific hypotheses about why LLMs fail. We find that modifications
which make explicit common inferences -- such as that looking at a transparent
object implies recognizing its contents -- preserve LLMs' performance. This
suggests that LLMs' failures on modified ToM tasks could result from a lack of
more general commonsense reasoning, rather than a failure to represent mental
states. We argue that SCALPEL could be helpful for explaining LLM successes and
failures in other cases.
