  In this paper, we introduce Watch-And-Help (WAH), a challenge for testing
social intelligence in agents. In WAH, an AI agent needs to help a human-like
agent perform a complex household task efficiently. To succeed, the AI agent
needs to i) understand the underlying goal of the task by watching a single
demonstration of the human-like agent performing the same task (social
perception), and ii) coordinate with the human-like agent to solve the task in
an unseen environment as fast as possible (human-AI collaboration). For this
challenge, we build VirtualHome-Social, a multi-agent household environment,
and provide a benchmark including both planning and learning based baselines.
We evaluate the performance of AI agents with the human-like agent as well as
with real humans using objective metrics and subjective user ratings.
Experimental results demonstrate that the proposed challenge and virtual
environment enable a systematic evaluation on the important aspects of machine
social intelligence at scale.
