  Although reinforcement learning has found widespread use in dense reward
settings, training autonomous agents with sparse rewards remains challenging.
To address this difficulty, prior work has shown promising results when using
not only task-specific demonstrations but also task-agnostic albeit somewhat
related demonstrations. In most cases, the available demonstrations are
distilled into an implicit prior, commonly represented via a single deep net.
Explicit priors in the form of a database that can be queried have also been
shown to lead to encouraging results. To better benefit from available
demonstrations, we develop a method to Combine Explicit and Implicit Priors
(CEIP). CEIP exploits multiple implicit priors in the form of normalizing flows
in parallel to form a single complex prior. Moreover, CEIP uses an effective
explicit retrieval and push-forward mechanism to condition the implicit priors.
In three challenging environments, we find the proposed CEIP method to improve
upon sophisticated state-of-the-art techniques.
