  We propose a new reinforcement learning based approach to designing
hierarchical linear quadratic regulator (LQR) controllers for heterogeneous
linear multi-agent systems with unknown state-space models and separated
control objectives. The separation arises from grouping the agents into
multiple non-overlapping groups, and defining the control goal as two distinct
objectives. The first objective aims to minimize a group-wise
block-decentralized LQR function that models group-level mission. The second
objective, on the other hand, tries to minimize an LQR function between the
average states (centroids) of the groups. Exploiting this separation, we
redefine the weighting matrices of the LQR functions in a way that they allow
us to decouple their respective algebraic Riccati equations. Thereafter, we
develop a reinforcement learning strategy that uses online measurements of the
agent states and the average states to learn the respective controllers based
on the approximate Riccati equations. Since the first controller is
block-decentralized and, therefore, can be learned in parallel, while the
second controller is reduced-dimensional due to averaging, the overall design
enjoys a significantly reduced learning time compared to centralized
reinforcement learning.
