  Policy networks are a central feature of deep reinforcement learning (RL)
algorithms for continuous control, enabling the estimation and sampling of
high-value actions. From the variational inference perspective on RL, policy
networks, when used with entropy or KL regularization, are a form of
\textit{amortized optimization}, optimizing network parameters rather than the
policy distributions directly. However, \textit{direct} amortized mappings can
yield suboptimal policy estimates and restricted distributions, limiting
performance and exploration. Given this perspective, we consider the more
flexible class of \textit{iterative} amortized optimizers. We demonstrate that
the resulting technique, iterative amortized policy optimization, yields
performance improvements over direct amortization on benchmark continuous
control tasks.
