  Exploration is a key challenge in Reinforcement Learning, especially in
long-horizon, deceptive and sparse-reward environments. For such applications,
population-based approaches have proven effective. Methods such as
Quality-Diversity deals with this by encouraging novel solutions and producing
a diversity of behaviours. However, these methods are driven by either
undirected sampling (i.e. mutations) or use approximated gradients (i.e.
Evolution Strategies) in the parameter space, which makes them highly
sample-inefficient. In this paper, we propose a model-based Quality-Diversity
approach. It extends existing QD methods to use gradients for efficient
exploitation and leverage perturbations in imagination for efficient
exploration. Our approach optimizes all members of a population simultaneously
to maintain both performance and diversity efficiently by leveraging the
effectiveness of QD algorithms as good data generators to train deep models. We
demonstrate that it maintains the divergent search capabilities of
population-based approaches on tasks with deceptive rewards while significantly
improving their sample efficiency and quality of solutions.
