  This paper studies a new variant of the stochastic multi-armed bandits
problem where auxiliary information about the arm rewards is available in the
form of control variates. In many applications like queuing and wireless
networks, the arm rewards are functions of some exogenous variables. The mean
values of these variables are known a priori from historical data and can be
used as control variates. Leveraging the theory of control variates, we obtain
mean estimates with smaller variance and tighter confidence bounds. We develop
an upper confidence bound based algorithm named UCB-CV and characterize the
regret bounds in terms of the correlation between rewards and control variates
when they follow a multivariate normal distribution. We also extend UCB-CV to
other distributions using resampling methods like Jackknifing and Splitting.
Experiments on synthetic problem instances validate performance guarantees of
the proposed algorithms.
