  Recent years have seen the application of deep reinforcement learning
techniques to cooperative multi-agent systems, with great empirical success.
However, given the lack of theoretical insight, it remains unclear what the
employed neural networks are learning, or how we should enhance their learning
power to address the problems on which they fail. In this work, we empirically
investigate the learning power of various network architectures on a series of
one-shot games. Despite their simplicity, these games capture many of the
crucial problems that arise in the multi-agent setting, such as an exponential
number of joint actions or the lack of an explicit coordination mechanism. Our
results extend those in [4] and quantify how well various approaches can
represent the requisite value functions, and help us identify the reasons that
can impede good performance, like sparsity of the values or too tight
coordination requirements.
