  Motivated by a number of real-world applications from domains like healthcare
and sustainable transportation, in this paper we study a scenario of repeated
principal-agent games within a multi-armed bandit (MAB) framework, where: the
principal gives a different incentive for each bandit arm, the agent picks a
bandit arm to maximize its own expected reward plus incentive, and the
principal observes which arm is chosen and receives a reward (different than
that of the agent) for the chosen arm. Designing policies for the principal is
challenging because the principal cannot directly observe the reward that the
agent receives for their chosen actions, and so the principal cannot directly
learn the expected reward using existing estimation techniques. As a result,
the problem of designing policies for this scenario, as well as similar ones,
remains mostly unexplored. In this paper, we construct a policy that achieves a
low regret (i.e., square-root regret up to a log factor) in this scenario for
the case where the agent has perfect-knowledge about its own expected rewards
for each bandit arm. We design our policy by first constructing an estimator
for the agent's expected reward for each bandit arm. Since our estimator uses
as data the sequence of incentives offered and subsequently chosen arms, the
principal's estimation can be regarded as an analogy of online inverse
optimization in MAB's. Next we construct a policy that we prove achieves a low
regret by deriving finite-sample concentration bounds for our estimator. We
conclude with numerical simulations demonstrating the applicability of our
policy to real-life setting from collaborative transportation planning.
