  As the scarce spectrum resource is becoming over-crowded, cognitive radios
(CRs) indicate great flexibility to improve the spectrum efficiency by
opportunistically accessing the authorized frequency bands. One of the critical
challenges for operating such radios in a network is how to efficiently
allocate transmission powers and frequency resource among the secondary users
(SUs) while satisfying the quality-of-service (QoS) constraints of the primary
users (PUs). In this paper, we focus on the non-cooperative power allocation
problem in cognitive wireless mesh networks (CogMesh) formed by a number of
clusters with the consideration of energy efficiency. Due to the SUs' selfish
and spontaneous properties, the problem is modeled as a stochastic learning
process. We first extend the single-agent Q-learning to a multi-user context,
and then propose a conjecture based multi-agent Qlearning algorithm to achieve
the optimal transmission strategies with only private and incomplete
information. An intelligent SU performs Q-function updates based on the
conjecture over the other SUs' stochastic behaviors. This learning algorithm
provably converges given certain restrictions that arise during learning
procedure. Simulation experiments are used to verify the performance of our
algorithm and demonstrate its effectiveness of improving the energy efficiency.
