  Graph-based representations and message-passing modular policies constitute
prominent approaches to tackling composable control problems in Reinforcement
Learning (RL). However, as shown by recent graph deep learning literature, such
local message-passing operators can create information bottlenecks and hinder
global coordination. The issue becomes more serious in tasks requiring
high-level planning. In this work, we propose a novel methodology, named Feudal
Graph Reinforcement Learning (FGRL), that addresses such challenges by relying
on hierarchical RL and a pyramidal message-passing architecture. In particular,
FGRL defines a hierarchy of policies where high-level commands are propagated
from the top of the hierarchy down through a layered graph structure. The
bottom layers mimic the morphology of the physical system, while the upper
layers correspond to higher-order sub-modules. The resulting agents are then
characterized by a committee of policies where actions at a certain level set
goals for the level below, thus implementing a hierarchical decision-making
structure that can naturally implement task decomposition. We evaluate the
proposed framework on a graph clustering problem and MuJoCo locomotion tasks;
simulation results show that FGRL compares favorably against relevant
baselines. Furthermore, an in-depth analysis of the command propagation
mechanism provides evidence that the introduced message-passing scheme favors
learning hierarchical decision-making policies.
