  Coagent networks formalize the concept of arbitrary networks of stochastic
agents that collaborate to take actions in a reinforcement learning
environment. Prominent examples of coagent networks in action include
approaches to hierarchical reinforcement learning (HRL), such as those using
options, which attempt to address the exploration exploitation trade-off by
introducing abstract actions at different levels by sequencing multiple
stochastic networks within the HRL agents. We first provide a unifying
perspective on the many diverse examples that fall under coagent networks. We
do so by formalizing the rules of execution in a coagent network, enabled by
the novel and intuitive idea of execution paths in a coagent network. Motivated
by parameter sharing in the hierarchical option-critic architecture, we revisit
the coagent network theory and achieve a much shorter proof of the policy
gradient theorem using our idea of execution paths, without any assumption on
how parameters are shared among coagents. We then generalize our setting and
proof to include the scenario where coagents act asynchronously. This new
perspective and theorem also lead to more mathematically accurate and
performant algorithms than those in the existing literature. Lastly, by running
nonstationary RL experiments, we survey the performance and properties of
different generalizations of option-critic models.
