  Heterogeneous Network (HetNet), where Small cell Base Stations (SBSs) are
densely deployed to offload traffic from macro Base Stations (BSs), is
identified as a key solution to meet the unprecedented mobile traffic demand.
The high density of SBSs are designed for peak traffic hours and consume an
unnecessarily large amount of energy during off-peak time. In this paper, we
propose a deep reinforcement-learning based SBS activation strategy that
activates the optimal subset of SBSs to significantly lower the energy
consumption without compromising the quality of service. In particular, we
formulate the SBS on/off switching problem into a Markov Decision Process that
can be solved by Actor Critic (AC) reinforcement learning methods. To avoid
prohibitively high computational and storage costs of conventional
tabular-based approaches, we propose to use deep neural networks to approximate
the policy and value functions in the AC approach. Moreover, to expedite the
training process, we adopt a Deep Deterministic Policy Gradient (DDPG) approach
together with a novel action refinement scheme. Through extensive numerical
simulations, we show that the proposed scheme greatly outperforms the existing
methods in terms of both energy efficiency and computational efficiency. We
also show that the proposed scheme can scale to large system with polynomial
complexities in both storage and computation.
