  Contextual bandit algorithms are sensitive to the estimation method of the
outcome model as well as the exploration method used, particularly in the
presence of rich heterogeneity or complex outcome models, which can lead to
difficult estimation problems along the path of learning. We develop algorithms
for contextual bandits with linear payoffs that integrate balancing methods
from the causal inference literature in their estimation to make it less prone
to problems of estimation bias. We provide the first regret bound analyses for
linear contextual bandits with balancing and show that our algorithms match the
state of the art theoretical guarantees. We demonstrate the strong practical
advantage of balanced contextual bandits on a large number of supervised
learning datasets and on a synthetic example that simulates model
misspecification and prejudice in the initial training data.
