  As multi-agent reinforcement learning (MARL) systems are increasingly
deployed throughout society, it is imperative yet challenging for users to
understand the emergent behaviors of MARL agents in complex environments. This
work presents an approach for generating policy-level contrastive explanations
for MARL to answer a temporal user query, which specifies a sequence of tasks
completed by agents with possible cooperation. The proposed approach encodes
the temporal query as a PCTL logic formula and checks if the query is feasible
under a given MARL policy via probabilistic model checking. Such explanations
can help reconcile discrepancies between the actual and anticipated multi-agent
behaviors. The proposed approach also generates correct and complete
explanations to pinpoint reasons that make a user query infeasible. We have
successfully applied the proposed approach to four benchmark MARL domains (up
to 9 agents in one domain). Moreover, the results of a user study show that the
generated explanations significantly improve user performance and satisfaction.
