  We consider the problem of learning to play a repeated contextual game with
unknown reward and unknown constraints functions. Such games arise in
applications where each agent's action needs to belong to a feasible set, but
the feasible set is a priori unknown. For example, in constrained multi-agent
reinforcement learning, the constraints on the agents' policies are a function
of the unknown dynamics and hence, are themselves unknown. Under kernel-based
regularity assumptions on the unknown functions, we develop a no-regret,
no-violation approach which exploits similarities among different reward and
constraint outcomes. The no-violation property ensures that the time-averaged
sum of constraint violations converges to zero as the game is repeated. We show
that our algorithm, referred to as c.z.AdaNormalGP, obtains kernel-dependent
regret bounds and that the cumulative constraint violations have sublinear
kernel-dependent upper bounds. In addition we introduce the notion of
constrained contextual coarse correlated equilibria (c.z.CCE) and show that
$\epsilon$-c.z.CCEs can be approached whenever players' follow a no-regret
no-violation strategy. Finally, we experimentally demonstrate the effectiveness
of c.z.AdaNormalGP on an instance of multi-agent reinforcement learning.
