  An optimal feedback controller for a given Markov decision process (MDP) can
in principle be synthesized by value or policy iteration. However, if the
system dynamics and the reward function are unknown, a learning agent must
discover an optimal controller via direct interaction with the environment.
Such interactive data gathering commonly leads to divergence towards dangerous
or uninformative regions of the state space unless additional regularization
measures are taken. Prior works proposed bounding the information loss measured
by the Kullback-Leibler (KL) divergence at every policy improvement step to
eliminate instability in the learning dynamics. In this paper, we consider a
broader family of $f$-divergences, and more concretely $\alpha$-divergences,
which inherit the beneficial property of providing the policy improvement step
in closed form at the same time yielding a corresponding dual objective for
policy evaluation. Such entropic proximal policy optimization view gives a
unified perspective on compatible actor-critic architectures. In particular,
common least-squares value function estimation coupled with advantage-weighted
maximum likelihood policy improvement is shown to correspond to the Pearson
$\chi^2$-divergence penalty. Other actor-critic pairs arise for various choices
of the penalty-generating function $f$. On a concrete instantiation of our
framework with the $\alpha$-divergence, we carry out asymptotic analysis of the
solutions for different values of $\alpha$ and demonstrate the effects of the
divergence function choice on common standard reinforcement learning problems.
