  Markov Decision Process (MDP) presents a mathematical framework to formulate
the learning processes of agents in reinforcement learning. MDP is limited by
the Markovian assumption that a reward only depends on the immediate state and
action. However, a reward sometimes depends on the history of states and
actions, which may result in the decision process in a non-Markovian
environment. In such environments, agents receive rewards via
temporally-extended behaviors sparsely, and the learned policies may be
similar. This leads the agents acquired with similar policies generally overfit
to the given task and can not quickly adapt to perturbations of environments.
To resolve this problem, this paper tries to learn the diverse policies from
the history of state-action pairs under a non-Markovian environment, in which a
policy dispersion scheme is designed for seeking diverse policy representation.
Specifically, we first adopt a transformer-based method to learn policy
embeddings. Then, we stack the policy embeddings to construct a dispersion
matrix to induce a set of diverse policies. Finally, we prove that if the
dispersion matrix is positive definite, the dispersed embeddings can
effectively enlarge the disagreements across policies, yielding a diverse
expression for the original policy embedding distribution. Experimental results
show that this dispersion scheme can obtain more expressive diverse policies,
which then derive more robust performance than recent learning baselines under
various learning environments.
