  The effectiveness of model-based versus model-free methods is a long-standing
question in reinforcement learning (RL). Motivated by recent empirical success
of RL on continuous control tasks, we study the sample complexity of popular
model-based and model-free algorithms on the Linear Quadratic Regulator (LQR).
We show that for policy evaluation, a simple model-based plugin method requires
asymptotically less samples than the classical least-squares temporal
difference (LSTD) estimator to reach the same quality of solution; the sample
complexity gap between the two methods can be at least a factor of state
dimension. For policy evaluation, we study a simple family of problem instances
and show that nominal (certainty equivalence principle) control also requires
several factors of state and input dimension fewer samples than the policy
gradient method to reach the same level of control performance on these
instances. Furthermore, the gap persists even when employing commonly used
baselines. To the best of our knowledge, this is the first theoretical result
which demonstrates a separation in the sample complexity between model-based
and model-free methods on a continuous control task.
