  Actor-critic algorithms address the dual goals of reinforcement learning
(RL), policy evaluation and improvement via two separate function
approximators. The practicality of this approach comes at the expense of
training instability, caused mainly by the destructive effect of the
approximation errors of the critic on the actor. We tackle this bottleneck by
employing an existing Probably Approximately Correct (PAC) Bayesian bound for
the first time as the critic training objective of the Soft Actor-Critic (SAC)
algorithm. We further demonstrate that online learning performance improves
significantly when a stochastic actor explores multiple futures by
critic-guided random search. We observe our resulting algorithm to compare
favorably against the state-of-the-art SAC implementation on multiple classical
control and locomotion tasks in terms of both sample efficiency and regret.
