  For optimal control of diffusions under several criteria, due to
computational or analytical reasons, many studies have a apriori assumed
control policies to be Lipschitz or smooth, often with no rigorous analysis on
whether this restriction entails loss. While optimality of Markov/stationary
Markov policies for expected finite horizon/infinite horizon
(discounted/ergodic) cost and cost-up-to-exit time optimal control problems can
be established under certain technical conditions, an optimal solution is
typically only measurable in the state (and time, if the horizon is finite)
with no apriori additional structural properties. In this paper, building on
our recent work [S. Pradhan and S. Y\"uksel, Continuity of cost in Borkar
control topology and implications on discrete space and time approximations for
controlled diffusions under several criteria, Electronic Journal of Probability
(2024)] establishing the regularity of optimal cost on the space of control
policies under the Borkar control topology for a general class of diffusions,
we establish near optimality of smooth/Lipschitz continuous policies for
optimal control under expected finite horizon, infinite horizon
discounted/average, and up-to-exit time cost criteria. Under mild assumptions,
we first show that smooth/Lipschitz continuous policies are dense in the space
of Markov/stationary Markov policies under the Borkar topology. Then utilizing
the continuity of optimal costs as a function of policies on the space of
Markov/stationary policies under the Borkar topology, we establish that optimal
policies can be approximated by smooth/Lipschitz continuous policies with
arbitrary precision. While our results are extensions of our recent work, the
practical significance of an explicit statement and accessible presentation
dedicated to Lipschitz and smooth policies, given their prominence in the
literature, motivates our current paper.
