  Linear quadratic regulator (LQR) is one of the most popular frameworks to
tackle continuous Markov decision process tasks. With its fundamental theory
and tractable optimal policy, LQR has been revisited and analyzed in recent
years, in terms of reinforcement learning scenarios such as the model-free or
model-based setting. In this paper, we introduce the \textit{Structured Policy
Iteration} (S-PI) for LQR, a method capable of deriving a structured linear
policy. Such a structured policy with (block) sparsity or low-rank can have
significant advantages over the standard LQR policy: more interpretable,
memory-efficient, and well-suited for the distributed setting. In order to
derive such a policy, we first cast a regularized LQR problem when the model is
known. Then, our Structured Policy Iteration (S-PI) algorithm, which takes a
policy evaluation step and a policy improvement step in an iterative manner,
can solve this regularized LQR efficiently. We further extend the S-PI
algorithm to the model-free setting where a smoothing procedure is adopted to
estimate the gradient. In both the known-model and model-free setting, we prove
convergence analysis under the proper choice of parameters. Finally, the
experiments demonstrate the advantages of S-PI in terms of balancing the LQR
performance and level of structure by varying the weight parameter.
