  Many reinforcement learning methods achieve great success in practice but
lack theoretical foundation. In this paper, we study the convergence analysis
on the problem of the Linear Quadratic Regulator (LQR). The global linear
convergence properties and sample complexities are established for several
popular algorithms such as the policy gradient algorithm, TD-learning and the
actor-critic (AC) algorithm. Our results show that the actor-critic algorithm
can reduce the sample complexity compared with the policy gradient algorithm.
Although our analysis is still preliminary, it explains the benefit of AC
algorithm in a certain sense.
