  Behavioural characterizations (BCs) of decision-making agents, or their
policies, are used to study outcomes of training algorithms and as part of the
algorithms themselves to encourage unique policies, match expert policy or
restrict changes to policy per update. However, previously presented solutions
are not applicable in general, either due to lack of expressive power,
computational constraint or constraints on the policy or environment.
Furthermore, many BCs rely on the actions of policies. We discuss and
demonstrate how these BCs can be misleading, especially in stochastic
environments, and propose a novel solution based on what states policies visit.
We run experiments to evaluate the quality of the proposed BC against baselines
and evaluate their use in studying training algorithms, novelty search and
trust-region policy optimization. The code is available at
https://github.com/miffyli/policy-supervectors.
