  Efficient exploration remains a challenging problem in reinforcement
learning, especially for those tasks where rewards from environments are
sparse. A commonly used approach for exploring such environments is to
introduce some "intrinsic" reward. In this work, we focus on model uncertainty
estimation as an intrinsic reward for efficient exploration. In particular, we
introduce an implicit generative modeling approach to estimate a Bayesian
uncertainty of the agent's belief of the environment dynamics. Each random draw
from our generative model is a neural network that instantiates the dynamic
function, hence multiple draws would approximate the posterior, and the
variance in the future prediction based on this posterior is used as an
intrinsic reward for exploration. We design a training algorithm for our
generative model based on the amortized Stein Variational Gradient Descent. In
experiments, we compare our implementation with state-of-the-art intrinsic
reward-based exploration approaches, including two recent approaches based on
an ensemble of dynamic models. In challenging exploration tasks, our implicit
generative model consistently outperforms competing approaches regarding data
efficiency in exploration.
