  We first show that the discounted cost, cost up to an exit time, and ergodic
cost involving controlled non-degenerate diffusions are continuous on the space
of stationary control policies when the policies are given a topology
introduced by Borkar [V. S. Borkar, A topology for Markov controls, Applied
Mathematics and Optimization 20 (1989), 55-62]. The same applies for finite
horizon problems when the control policies are markov and the topology is
revised to include time also as a parameter. We then establish that finite
action/piecewise constant stationary policies are dense in the space of
stationary Markov policies under this topology. Using the above mentioned
continuity and denseness results we establish that finite action/piecewise
constant policies approximate optimal stationary policies with arbitrary
precision. This gives rise to the applicability of many numerical methods such
as policy iteration and stochastic learning methods for discounted cost, cost
up to an exit time, and ergodic cost optimal control problems in
continuous-time. For the finite-horizon setup, we establish additionally near
optimality of time-discretized policies by an analogous argument. We thus
present a unified and concise approach for approximations directly applicable
under several commonly adopted cost criteria.
