  Designing the optimal linear quadratic regulator (LQR) for a large-scale
multi-agent system (MAS) is time-consuming since it involves solving a
large-size matrix Riccati equation. The situation is further exasperated when
the design needs to be done in a model-free way using schemes such as
reinforcement learning (RL). To reduce this computational complexity, we
decompose the large-scale LQR design problem into multiple smaller-size LQR
design problems. We consider the objective function to be specified over an
undirected graph, and cast the decomposition as a graph clustering problem. The
graph is decomposed into two parts, one consisting of independent clusters of
connected components, and the other containing edges that connect different
clusters. Accordingly, the resulting controller has a hierarchical structure,
consisting of two components. The first component optimizes the performance of
each independent cluster by solving the smaller-size LQR design problem in a
model-free way using an RL algorithm. The second component accounts for the
objective coupling different clusters, which is achieved by solving a least
squares problem in one shot. Although suboptimal, the hierarchical controller
adheres to a particular structure as specified by inter-agent couplings in the
objective function and by the decomposition strategy. Mathematical formulations
are established to find a decomposition that minimizes the number of required
communication links or reduces the optimality gap. Numerical simulations are
provided to highlight the pros and cons of the proposed designs.
