  The escalating debate on AI's capabilities warrants developing reliable
metrics to assess machine "intelligence". Recently, many anecdotal examples
were used to suggest that newer large language models (LLMs) like ChatGPT and
GPT-4 exhibit Neural Theory-of-Mind (N-ToM); however, prior work reached
conflicting conclusions regarding those abilities. We investigate the extent of
LLMs' N-ToM through an extensive evaluation on 6 tasks and find that while LLMs
exhibit certain N-ToM abilities, this behavior is far from being robust. We
further examine the factors impacting performance on N-ToM tasks and discover
that LLMs struggle with adversarial examples, indicating reliance on shallow
heuristics rather than robust ToM abilities. We caution against drawing
conclusions from anecdotal examples, limited benchmark testing, and using
human-designed psychological tests to evaluate models.
