  Multi-agent reinforcement learning (RL) often struggles to ensure the safe
behaviours of the learning agents, and therefore it is generally not adapted to
safety-critical applications. To address this issue, we present a methodology
that combines formal verification with (deep) RL algorithms to guarantee the
satisfaction of formally-specified safety constraints both in training and
testing. The approach we propose expresses the constraints to verify in
Probabilistic Computation Tree Logic (PCTL) and builds an abstract
representation of the system to reduce the complexity of the verification step.
This abstract model allows for model checking techniques to identify a set of
abstract policies that meet the safety constraints expressed in PCTL. Then, the
agents' behaviours are restricted according to these safe abstract policies. We
provide formal guarantees that by using this method, the actions of the agents
always meet the safety constraints, and provide a procedure to generate an
abstract model automatically. We empirically evaluate and show the
effectiveness of our method in a multi-agent environment.
