  Multi-agent reinforcement learning (MARL) methods struggle with the
non-stationarity of multi-agent systems and fail to adaptively learn online
when tested with novel agents. Here, we leverage large language models (LLMs)
to create an autonomous agent that can handle these challenges. Our agent,
Hypothetical Minds, consists of a cognitively-inspired architecture, featuring
modular components for perception, memory, and hierarchical planning over two
levels of abstraction. We introduce the Theory of Mind module that scaffolds
the high-level planning process by generating hypotheses about other agents'
strategies in natural language. It then evaluates and iteratively refines these
hypotheses by reinforcing hypotheses that make correct predictions about the
other agents' behavior. Hypothetical Minds significantly improves performance
over previous LLM-agent and RL baselines on a range of competitive, mixed
motive, and collaborative domains in the Melting Pot benchmark, including both
dyadic and population-based environments. Additionally, comparisons against
LLM-agent baselines and ablations reveal the importance of hypothesis
evaluation and refinement for succeeding on complex scenarios.
