  Device-to-device (D2D) communication has been recognized as a promising
technique to improve spectrum efficiency. However, D2D transmission as an
underlay causes severe interference, which imposes a technical challenge to
spectrum allocation. Existing centralized schemes require global information,
which can cause serious signaling overhead. While existing distributed solution
requires frequent information exchange between users and cannot achieve global
optimization. In this paper, a distributed spectrum allocation framework based
on multi-agent deep reinforcement learning is proposed, named Neighbor-Agent
Actor Critic (NAAC). NAAC uses neighbor users' historical information for
centralized training but is executed distributedly without that information,
which not only has no signal interaction during execution, but also utilizes
cooperation between users to further optimize system performance. The
simulation results show that the proposed framework can effectively reduce the
outage probability of cellular links, improve the sum rate of D2D links and
have good convergence.
