  While Large Language Models (LLMs) have demonstrated impressive
accomplishments in both reasoning and planning, their abilities in multi-agent
collaborations remains largely unexplored. This study evaluates LLM-based
agents in a multi-agent cooperative text game with Theory of Mind (ToM)
inference tasks, comparing their performance with Multi-Agent Reinforcement
Learning (MARL) and planning-based baselines. We observed evidence of emergent
collaborative behaviors and high-order Theory of Mind capabilities among
LLM-based agents. Our results reveal limitations in LLM-based agents' planning
optimization due to systematic failures in managing long-horizon contexts and
hallucination about the task state. We explore the use of explicit belief state
representations to mitigate these issues, finding that it enhances task
performance and the accuracy of ToM inferences for LLM-based agents.
