  We present efficient algorithms for the problem of contextual bandits with
i.i.d. covariates, an arbitrary sequence of rewards, and an arbitrary class of
policies. Our algorithm BISTRO requires d calls to the empirical risk
minimization (ERM) oracle per round, where d is the number of actions. The
method uses unlabeled data to make the problem computationally simple. When the
ERM problem itself is computationally hard, we extend the approach by employing
multiplicative approximation algorithms for the ERM. The integrality gap of the
relaxation only enters in the regret bound rather than the benchmark. Finally,
we show that the adversarial version of the contextual bandit problem is
learnable (and efficient) whenever the full-information supervised online
learning problem has a non-trivial regret guarantee (and efficient).
