  We introduce the Laser Learning Environment (LLE), a collaborative
multi-agent reinforcement learning environment in which coordination is
central. In LLE, agents depend on each other to make progress
(interdependence), must jointly take specific sequences of actions to succeed
(perfect coordination), and accomplishing those joint actions does not yield
any intermediate reward (zero-incentive dynamics). The challenge of such
problems lies in the difficulty of escaping state space bottlenecks caused by
interdependence steps since escaping those bottlenecks is not rewarded. We test
multiple state-of-the-art value-based MARL algorithms against LLE and show that
they consistently fail at the collaborative task because of their inability to
escape state space bottlenecks, even though they successfully achieve perfect
coordination. We show that Q-learning extensions such as prioritized experience
replay and n-steps return hinder exploration in environments with
zero-incentive dynamics, and find that intrinsic curiosity with random network
distillation is not sufficient to escape those bottlenecks. We demonstrate the
need for novel methods to solve this problem and the relevance of LLE as
cooperative MARL benchmark.
