  This paper investigates learning-based caching in small-cell networks (SCNs)
when user preference is unknown. The goal is to optimize the cache placement in
each small base station (SBS) for minimizing the system long-term transmission
delay. We model this sequential multi-agent decision making problem in a
multi-agent multi-armed bandit (MAMAB) perspective. Rather than estimating user
preference first and then optimizing the cache strategy, we propose several
MAMAB-based algorithms to directly learn the cache strategy online in both
stationary and non-stationary environment. In the stationary environment, we
first propose two high-complexity agent-based collaborative MAMAB algorithms
with performance guarantee. Then we propose a low-complexity distributed MAMAB
which ignores the SBS coordination. To achieve a better balance between SBS
coordination gain and computational complexity, we develop an edge-based
collaborative MAMAB with the coordination graph edge-based reward assignment
method. In the non-stationary environment, we modify the MAMAB-based algorithms
proposed in the stationary environment by proposing a practical initialization
method and designing new perturbed terms to adapt to the dynamic environment.
Simulation results are provided to validate the effectiveness of our proposed
algorithms. The effects of different parameters on caching performance are also
discussed.
