  In this paper we consider discrete time stochastic optimal control problems
over infinite and finite time horizons. We show that for a large class of such
problems the Taylor polynomials of the solutions to the associated Dynamic
Programming Equations can be computed degree by degree.
