  Model-free deep reinforcement learning has achieved great success in many
domains, such as video games, recommendation systems and robotic control tasks.
In continuous control tasks, widely used policies with Gaussian distributions
results in ineffective exploration of environments and limited performance of
algorithms in many cases. In this paper, we propose a density-free off-policy
algorithm, Generative Actor-Critic(GAC), using the push-forward model to
increase the expressiveness of policies, which also includes an entropy-like
technique, MMD-entropy regularizer, to balance the exploration and
exploitation. Additionnally, we devise an adaptive mechanism to automatically
scale this regularizer, which further improves the stability and robustness of
GAC. The experiment results show that push-forward policies possess desirable
features, such as multi-modality, which can improve the efficiency of
exploration and asymptotic performance of algorithms obviously.
