  Entropy regularization is commonly used to improve policy optimization in
reinforcement learning. It is believed to help with \emph{exploration} by
encouraging the selection of more stochastic policies. In this work, we analyze
this claim using new visualizations of the optimization landscape based on
randomly perturbing the loss function. We first show that even with access to
the exact gradient, policy optimization is difficult due to the geometry of the
objective function. Then, we qualitatively show that in some environments, a
policy with higher entropy can make the optimization landscape smoother,
thereby connecting local optima and enabling the use of larger learning rates.
This paper presents new tools for understanding the optimization landscape,
shows that policy entropy serves as a regularizer, and highlights the challenge
of designing general-purpose policy optimization algorithms.
