  Vision-based reinforcement learning (RL) is a promising approach to solve
control tasks involving images as the main observation. State-of-the-art RL
algorithms still struggle in terms of sample efficiency, especially when using
image observations. This has led to increased attention on integrating state
representation learning (SRL) techniques into the RL pipeline. Work in this
field demonstrates a substantial improvement in sample efficiency among other
benefits. However, to take full advantage of this paradigm, the quality of
samples used for training plays a crucial role. More importantly, the diversity
of these samples could affect the sample efficiency of vision-based RL, but
also its generalization capability. In this work, we present an approach to
improve sample diversity for state representation learning. Our method enhances
the exploration capability of RL algorithms, by taking advantage of the SRL
setup. Our experiments show that our proposed approach boosts the visitation of
problematic states, improves the learned state representation, and outperforms
the baselines for all tested environments. These results are most apparent for
environments where the baseline methods struggle. Even in simple environments,
our method stabilizes the training, reduces the reward variance, and promotes
sample efficiency.
