  This paper investigates energy efficiency for two-tier femtocell networks
through combining game theory and stochastic learning. With the Stackelberg
game formulation, a hierarchical reinforcement learning framework is applied to
study the joint average utility maximization of macrocells and femtocells
subject to the minimum signal-to-interference-plus-noise-ratio requirements.
The macrocells behave as the leaders and the femtocells are followers during
the learning procedure. At each time step, the leaders commit to dynamic
strategies based on the best responses of the followers, while the followers
compete against each other with no further information but the leaders'
strategy information. In this paper, we propose two learning algorithms to
schedule each cell's stochastic power levels, leading by the macrocells.
Numerical experiments are presented to validate the proposed studies and show
that the two learning algorithms substantially improve the energy efficiency of
the femtocell networks.
