  Introducing cooperative coded caching into small cell networks is a promising
approach to reducing traffic loads. By encoding content via maximum distance
separable (MDS) codes, coded fragments can be collectively cached at small-cell
base stations (SBSs) to enhance caching efficiency. However, content popularity
is usually time-varying and unknown in practice. As a result, cache contents
are anticipated to be intelligently updated by taking into account limited
caching storage and interactive impacts among SBSs. In response to these
challenges, we propose a multi-agent deep reinforcement learning (DRL)
framework to intelligently update cache contents in dynamic environments. With
the goal of minimizing long-term expected fronthaul traffic loads, we first
model dynamic coded caching as a cooperative multi-agent Markov decision
process. Owing to MDS coding, the resulting decision-making falls into a class
of constrained reinforcement learning problems with continuous decision
variables. To deal with this difficulty, we custom-build a novel DRL algorithm
by embedding homotopy optimization into a deep deterministic policy gradient
formalism. Next, to empower the caching framework with an effective trade-off
between complexity and performance, we propose centralized, partially and fully
decentralized caching controls by applying the derived DRL approach. Simulation
results demonstrate the superior performance of the proposed multi-agent
framework.
