  Adaptive dynamic programming (ADP) for stochastic linear quadratic regulation
(LQR) demands the precise computation of stochastic integrals during policy
iteration (PI). In a fully model-free problem setting, this computation can
only be approximated by state samples collected at discrete time points using
computational methods such as the canonical Euler-Maruyama method. Our research
reveals a critical phenomenon: the sampling period can significantly impact
control performance. This impact is due to the fact that computational errors
introduced in each step of PI can significantly affect the algorithm's
convergence behavior, which in turn influences the resulting control policy. We
draw a parallel between PI and Newton's method applied to the Ricatti equation
to elucidate how the computation impacts control. In this light, the
computational error in each PI step manifests itself as an extra error term in
each step of Newton's method, with its upper bound proportional to the
computational error. Furthermore, we demonstrate that the convergence rate for
ADP in stochastic LQR problems using the Euler-Maruyama method is O(h), with h
being the sampling period. A sensorimotor control task finally validates these
theoretical findings.
