  Policy gradient methods have enabled deep reinforcement learning (RL) to
approach challenging continuous control problems, even when the underlying
systems involve highly nonlinear dynamics that generate complex non-smooth
optimization landscapes. We develop a rigorous framework for understanding how
policy gradient methods mollify non-smooth optimization landscapes to enable
effective policy search, as well as the downside of it: while making the
objective function smoother and easier to optimize, the stochastic objective
deviates further from the original problem. We demonstrate the equivalence
between policy gradient methods and solving backward heat equations. Following
the ill-posedness of backward heat equations from PDE theory, we present a
fundamental challenge to the use of policy gradient under stochasticity.
Moreover, we make the connection between this limitation and the uncertainty
principle in harmonic analysis to understand the effects of exploration with
stochastic policies in RL. We also provide experimental results to illustrate
both the positive and negative aspects of mollification effects in practice.
