  In heterogeneous networks (HetNets), the overlap of small cells and the macro
cell causes severe cross-tier interference. Although there exist some
approaches to address this problem, they usually require global channel state
information, which is hard to obtain in practice, and get the sub-optimal power
allocation policy with high computational complexity. To overcome these
limitations, we propose a multi-agent deep reinforcement learning (MADRL) based
power control scheme for the HetNet, where each access point makes power
control decisions independently based on local information. To promote
cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm
for MADRL systems. By introducing regularization terms in the loss function,
each agent tends to choose an experienced action with high reward when
revisiting a state, and thus the policy updating speed slows down. In this way,
an agent's policy can be learned by other agents more easily, resulting in a
more efficient collaboration process. We then implement the proposed PQL in the
considered HetNet and compare it with other distributed-training-and-execution
(DTE) algorithms. Simulation results show that our proposed PQL can learn the
desired power control policy from a dynamic environment where the locations of
users change episodically and outperform existing DTE MADRL algorithms.
