  We consider a repeated sequential game between a learner, who plays first,
and an opponent who responds to the chosen action. We seek to design strategies
for the learner to successfully interact with the opponent. While most previous
approaches consider known opponent models, we focus on the setting in which the
opponent's model is unknown. To this end, we use kernel-based regularity
assumptions to capture and exploit the structure in the opponent's response. We
propose a novel algorithm for the learner when playing against an adversarial
sequence of opponents. The algorithm combines ideas from bilevel optimization
and online learning to effectively balance between exploration (learning about
the opponent's model) and exploitation (selecting highly rewarding actions for
the learner). Our results include algorithm's regret guarantees that depend on
the regularity of the opponent's response and scale sublinearly with the number
of game rounds. Moreover, we specialize our approach to repeated Stackelberg
games, and empirically demonstrate its effectiveness in a traffic routing and
wildlife conservation task
