  When learning from a batch of logged bandit feedback, the discrepancy between
the policy to be learned and the off-policy training data imposes statistical
and computational challenges. Unlike classical supervised learning and online
learning settings, in batch contextual bandit learning, one only has access to
a collection of logged feedback from the actions taken by a historical policy,
and expect to learn a policy that takes good actions in possibly unseen
contexts. Such a batch learning setting is ubiquitous in online and interactive
systems, such as ad platforms and recommendation systems. Existing approaches
based on inverse propensity weights, such as Inverse Propensity Scoring (IPS)
and Policy Optimizer for Exponential Models (POEM), enjoy unbiasedness but
often suffer from large mean squared error. In this work, we introduce a new
approach named Maximum Likelihood Inverse Propensity Scoring (MLIPS) for batch
learning from logged bandit feedback. Instead of using the given historical
policy as the proposal in inverse propensity weights, we estimate a maximum
likelihood surrogate policy based on the logged action-context pairs, and then
use this surrogate policy as the proposal. We prove that MLIPS is
asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared
error than IPS. Such an error reduction phenomenon is somewhat surprising as
the estimated surrogate policy is less accurate than the given historical
policy. Results on multi-label classification problems and a large- scale ad
placement dataset demonstrate the empirical effectiveness of MLIPS.
Furthermore, the proposed surrogate policy technique is complementary to
existing error reduction techniques, and when combined, is able to consistently
boost the performance of several widely used approaches.
