  We present an algorithm for learning an approximate action-value soft
Q-function in the relative entropy regularised reinforcement learning setting,
for which an optimal improved policy can be recovered in closed form. We use
recent advances in normalising flows for parametrising the policy together with
a learned value-function; and show how this combination can be used to
implicitly represent Q-values of an arbitrary policy in continuous action
space. Using simple temporal difference learning on the Q-values then leads to
a unified objective for policy and value learning. We show how this approach
considerably simplifies standard Actor-Critic off-policy algorithms, removing
the need for a policy optimisation step. We perform experiments on a range of
established reinforcement learning benchmarks, demonstrating that our approach
allows for complex, multimodal policy distributions in continuous action
spaces, while keeping the process of sampling from the policy both fast and
exact.
