  Current research directions in deep reinforcement learning include bridging
the simulation-reality gap, improving sample efficiency of experiences in
distributed multi-agent reinforcement learning, together with the development
of robust methods against adversarial agents in distributed learning, among
many others. In this work, we are particularly interested in analyzing how
multi-agent reinforcement learning can bridge the gap to reality in distributed
multi-robot systems where the operation of the different robots is not
necessarily homogeneous. These variations can happen due to sensing mismatches,
inherent errors in terms of calibration of the mechanical joints, or simple
differences in accuracy. While our results are simulation-based, we introduce
the effect of sensing, calibration, and accuracy mismatches in distributed
reinforcement learning with proximal policy optimization (PPO). We discuss on
how both the different types of perturbances and how the number of agents
experiencing those perturbances affect the collaborative learning effort. The
simulations are carried out using a Kuka arm model in the Bullet physics
engine. This is, to the best of our knowledge, the first work exploring the
limitations of PPO in multi-robot systems when considering that different
robots might be exposed to different environments where their sensors or
actuators have induced errors. With the conclusions of this work, we set the
initial point for future work on designing and developing methods to achieve
robust reinforcement learning on the presence of real-world perturbances that
might differ within a multi-robot system.
