  Recent years have seen significant advances in explainable AI as the need to
understand deep learning models has gained importance with the increased
emphasis on trust and ethics in AI. Comprehensible models for sequential
decision tasks are a particular challenge as they require understanding not
only individual predictions but a series of predictions that interact with
environmental dynamics. We present a framework for learning comprehensible
models of sequential decision tasks in which agent strategies are characterized
using temporal logic formulas. Given a set of agent traces, we first cluster
the traces using a novel embedding method that captures frequent action
patterns. We then search for logical formulas that explain the agent strategies
in the different clusters. We evaluate our framework on combat scenarios in
StarCraft II (SC2), using traces from a handcrafted expert policy and a trained
reinforcement learning agent. We implemented a feature extractor for SC2
environments that extracts traces as sequences of high-level features
describing both the state of the environment and the agent's local behavior
from agent replays. We further designed a visualization tool depicting the
movement of units in the environment that helps understand how different task
conditions lead to distinct agent behavior patterns in each trace cluster.
Experimental results show that our framework is capable of separating agent
traces into distinct groups of behaviors for which our approach to strategy
inference produces consistent, meaningful, and easily understood strategy
descriptions.
