  Uncertainty quantification has been extensively used as a means to achieve
efficient directed exploration in Reinforcement Learning (RL). However,
state-of-the-art methods for continuous actions still suffer from high sample
complexity requirements. Indeed, they either completely lack strategies for
propagating the epistemic uncertainty throughout the updates, or they mix it
with aleatoric uncertainty while learning the full return distribution (e.g.,
distributional RL). In this paper, we propose Wasserstein Actor-Critic (WAC),
an actor-critic architecture inspired by the recent Wasserstein Q-Learning
(WQL) \citep{wql}, that employs approximate Q-posteriors to represent the
epistemic uncertainty and Wasserstein barycenters for uncertainty propagation
across the state-action space. WAC enforces exploration in a principled way by
guiding the policy learning process with the optimization of an upper bound of
the Q-value estimates. Furthermore, we study some peculiar issues that arise
when using function approximation, coupled with the uncertainty estimation, and
propose a regularized loss for the uncertainty estimation. Finally, we evaluate
our algorithm on standard MujoCo tasks as well as suite of continuous-actions
domains, where exploration is crucial, in comparison with state-of-the-art
baselines.
