  This paper introduces a full solution for decentralized routing in Low Earth
Orbit satellite constellations based on continual Deep Reinforcement Learning
(DRL). This requires addressing multiple challenges, including the partial
knowledge at the satellites and their continuous movement, and the time-varying
sources of uncertainty in the system, such as traffic, communication links, or
communication buffers. We follow a multi-agent approach, where each satellite
acts as an independent decision-making agent, while acquiring a limited
knowledge of the environment based on the feedback received from the nearby
agents. The solution is divided into two phases. First, an offline learning
phase relies on decentralized decisions and a global Deep Neural Network (DNN)
trained with global experiences. Then, the online phase with local, on-board,
and pre-trained DNNs requires continual learning to evolve with the
environment, which can be done in two different ways: (1) Model anticipation,
where the predictable conditions of the constellation are exploited by each
satellite sharing local model with the next satellite; and (2) Federated
Learning (FL), where each agent's model is merged first at the cluster level
and then aggregated in a global Parameter Server. The results show that,
without high congestion, the proposed Multi-Agent DRL framework achieves the
same E2E performance as a shortest-path solution, but the latter assumes
intensive communication overhead for real-time network-wise knowledge of the
system at a centralized node, whereas ours only requires limited feedback
exchange among first neighbour satellites. Importantly, our solution adapts
well to congestion conditions and exploits less loaded paths. Moreover, the
divergence of models over time is easily tackled by the synergy between
anticipation, applied in short-term alignment, and FL, utilized for long-term
alignment.
