  We propose an online learning algorithm for a class of machine learning
models under a separable stochastic approximation framework. The essence of our
idea lies in the observation that certain parameters in the models are easier
to optimize than others. In this paper, we focus on models where some
parameters have a linear nature, which is common in machine learning. In one
routine of the proposed algorithm, the linear parameters are updated by the
recursive least squares (RLS) algorithm, which is equivalent to a stochastic
Newton method; then, based on the updated linear parameters, the nonlinear
parameters are updated by the stochastic gradient method (SGD). The proposed
algorithm can be understood as a stochastic approximation version of block
coordinate gradient descent approach in which one part of the parameters is
updated by a second-order SGD method while the other part is updated by a
first-order SGD. Global convergence of the proposed online algorithm for
non-convex cases is established in terms of the expected violation of a
first-order optimality condition. Numerical experiments show that the proposed
method accelerates convergence significantly and produces more robust training
and test performance when compared to other popular learning algorithms.
Moreover, our algorithm is less sensitive to the learning rate and outperforms
the recently proposed slimTrain algorithm (Newman et al., 2022). The code has
been uploaded to GitHub for validation.
