  Learning effective contextual-bandit policies from past actions of a deployed
system is highly desirable in many settings (e.g. voice assistants,
recommendation, search), since it enables the reuse of large amounts of log
data. State-of-the-art methods for such off-policy learning, however, are based
on inverse propensity score (IPS) weighting. A key theoretical requirement of
IPS weighting is that the policy that logged the data has "full support", which
typically translates into requiring non-zero probability for any action in any
context. Unfortunately, many real-world systems produce support deficient data,
especially when the action space is large, and we show how existing methods can
fail catastrophically. To overcome this gap between theory and applications, we
identify three approaches that provide various guarantees for IPS-based
learning despite the inherent limitations of support-deficient data:
restricting the action space, reward extrapolation, and restricting the policy
space. We systematically analyze the statistical and computational properties
of these three approaches, and we empirically evaluate their effectiveness. In
addition to providing the first systematic analysis of support-deficiency in
contextual-bandit learning, we conclude with recommendations that provide
practical guidance.
