  Communication is crucial in multi-agent reinforcement learning when agents
are not able to observe the full state of the environment. The most common
approach to allow learned communication between agents is the use of a
differentiable communication channel that allows gradients to flow between
agents as a form of feedback. However, this is challenging when we want to use
discrete messages to reduce the message size, since gradients cannot flow
through a discrete communication channel. Previous work proposed methods to
deal with this problem. However, these methods are tested in different
communication learning architectures and environments, making it hard to
compare them. In this paper, we compare several state-of-the-art discretization
methods as well as a novel approach. We do this comparison in the context of
communication learning using gradients from other agents and perform tests on
several environments. In addition, we present COMA-DIAL, a communication
learning approach based on DIAL and COMA extended with learning rate scaling
and adapted exploration. Using COMA-DIAL allows us to perform experiments on
more complex environments. Our results show that the novel ST-DRU method,
proposed in this paper, achieves the best results out of all discretization
methods across the different environments. It achieves the best or close to the
best performance in each of the experiments and is the only method that does
not fail on any of the tested environments.
