  The actor-critic (AC) reinforcement learning algorithms have been the
powerhouse behind many challenging applications. Nevertheless, its convergence
is fragile in general. To study its instability, existing works mostly consider
the uncommon double-loop variant or basic models with finite state and action
space. We investigate the more practical single-sample two-timescale AC for
solving the canonical linear quadratic regulator (LQR) problem, where the actor
and the critic update only once with a single sample in each iteration on an
unbounded continuous state and action space. Existing analysis cannot conclude
the convergence for such a challenging case. We develop a new analysis
framework that allows establishing the global convergence to an
$\epsilon$-optimal solution with at most an $\mathcal{O}(\epsilon^{-2.5})$
sample complexity. To our knowledge, this is the first finite-time convergence
analysis for the single sample two-timescale AC for solving LQR with global
optimality. The sample complexity improves those of other variants by orders,
which sheds light on the practical wisdom of single sample algorithms. We also
further validate our theoretical findings via comprehensive simulation
comparisons.
