  Learning optimal policies in sparse rewards settings is difficult as the
learning agent has little to no feedback on the quality of its actions. In
these situations, a good strategy is to focus on exploration, hopefully leading
to the discovery of a reward signal to improve on. A learning algorithm capable
of dealing with this kind of settings has to be able to (1) explore possible
agent behaviors and (2) exploit any possible discovered reward. Efficient
exploration algorithms have been proposed that require to define a behavior
space, that associates to an agent its resulting behavior in a space that is
known to be worth exploring. The need to define this space is a limitation of
these algorithms. In this work, we introduce STAX, an algorithm designed to
learn a behavior space on-the-fly and to explore it while efficiently
optimizing any reward discovered. It does so by separating the exploration and
learning of the behavior space from the exploitation of the reward through an
alternating two-steps process. In the first step, STAX builds a repertoire of
diverse policies while learning a low-dimensional representation of the
high-dimensional observations generated during the policies evaluation. In the
exploitation step, emitters are used to optimize the performance of the
discovered rewarding solutions. Experiments conducted on three different sparse
reward environments show that STAX performs comparably to existing baselines
while requiring much less prior information about the task as it autonomously
builds the behavior space.
