  Stochastic approximation (SA) is a powerful class of iterative algorithms for
nonlinear root-finding that can be used for minimizing a loss function,
$L(\boldsymbol{\theta})$, with respect to a parameter vector
$\boldsymbol{\theta}$, when only noisy observations of $L(\boldsymbol{\theta})$
or its gradient are available (through the natural connection between
root-finding and minimization); SA algorithms can be thought of as stochastic
line search methods where the entire parameter vector is updated at each
iteration. The cyclic approach to SA is a variant of SA procedures where
$\boldsymbol{\theta}$ is divided into multiple subvectors that are updated one
at a time in a cyclic manner. This dissertation focuses on studying the
asymptotic properties of cyclic SA and of the generalized cyclic SA (GCSA)
algorithm, a variant of cyclic SA where the subvector to update may be selected
according to a random variable or according to a predetermined pattern, and
where the noisy update direction can be based on the updates of any SA
algorithm (e.g., stochastic gradient, Kiefer--Wolfowitz, or simultaneous
perturbation SA). The convergence of GCSA, asymptotic normality of GCSA
(related to rate of convergence), and efficiency of GCSA relative to its
non-cyclic counterpart are investigated both analytically and numerically.
Specifically, conditions are obtained for the convergence with probability one
of the GCSA iterates and for the asymptotic normality of the normalized
iterates of a special case of GCSA. Further, an analytic expression is given
for the asymptotic relative efficiency (when efficiency is defined in terms of
mean squared error) between a special case of GCSA and its non-cyclic
counterpart. Finally, an application of the cyclic SA scheme to a multi-agent
stochastic optimization problem is investigated. This dissertation also
contains two appendices.
