  One of the efficient solutions of improving coverage and increasing capacity
in cellular networks is the deployment of femtocells. As the cellular networks
are becoming more complex, energy consumption of whole network infrastructure
is becoming important in terms of both operational costs and environmental
impacts. This paper investigates energy efficiency of two-tier femtocell
networks through combining game theory and stochastic learning. With the
Stackelberg game formulation, a hierarchical reinforcement learning framework
is applied for studying the joint expected utility maximization of macrocells
and femtocells subject to the minimum signal-to-interference-plus-noise-ratio
requirements. In the learning procedure, the macrocells act as leaders and the
femtocells are followers. At each time step, the leaders commit to dynamic
strategies based on the best responses of the followers, while the followers
compete against each other with no further information but the leaders'
transmission parameters. In this paper, we propose two reinforcement learning
based intelligent algorithms to schedule each cell's stochastic power levels.
Numerical experiments are presented to validate the investigations. The results
show that the two learning algorithms substantially improve the energy
efficiency of the femtocell networks.
