  Autonomous agents must learn to collaborate. It is not scalable to develop a
new centralized agent every time a task's difficulty outpaces a single agent's
abilities. While multi-agent collaboration research has flourished in
gridworld-like environments, relatively little work has considered visually
rich domains. Addressing this, we introduce the novel task FurnMove in which
agents work together to move a piece of furniture through a living room to a
goal. Unlike existing tasks, FurnMove requires agents to coordinate at every
timestep. We identify two challenges when training agents to complete FurnMove:
existing decentralized action sampling procedures do not permit expressive
joint action policies and, in tasks requiring close coordination, the number of
failed actions dominates successful actions. To confront these challenges we
introduce SYNC-policies (synchronize your actions coherently) and CORDIAL
(coordination loss). Using SYNC-policies and CORDIAL, our agents achieve a 58%
completion rate on FurnMove, an impressive absolute gain of 25 percentage
points over competitive decentralized baselines. Our dataset, code, and
pretrained models are available at https://unnat.github.io/cordial-sync .
