  We study constrained general-sum stochastic games with unknown Markovian
dynamics. A distributed constrained no-regret Q-learning scheme (CNRQ) is
presented to guarantee convergence to the set of stationary correlated
equilibria of the game. Prior art addresses the unconstrained case only, is
structured with nested control loops, and has no convergence result. CNRQ is
cast as a single-loop three-timescale asynchronous stochastic approximation
algorithm with set-valued update increments. A rigorous convergence analysis
with differential inclusion arguments is given which draws on recent extensions
of the theory of stochastic approximation to the case of asynchronous recursive
inclusions with set-valued mean fields. Numerical results are given for the
exemplary application of CNRQ to decentralized resource control in
heterogeneous wireless networks (HetNets).
