  We study sequential decision making in environments where rewards are only
partially observed, but can be modeled as a function of observed contexts and
the chosen action by the decision maker. This setting, known as contextual
bandits, encompasses a wide variety of applications such as health care,
content recommendation and Internet advertising. A central task is evaluation
of a new policy given historic data consisting of contexts, actions and
received rewards. The key challenge is that the past data typically does not
faithfully represent proportions of actions taken by a new policy. Previous
approaches rely either on models of rewards or models of the past policy. The
former are plagued by a large bias whereas the latter have a large variance. In
this work, we leverage the strengths and overcome the weaknesses of the two
approaches by applying the doubly robust estimation technique to the problems
of policy evaluation and optimization. We prove that this approach yields
accurate value estimates when we have either a good (but not necessarily
consistent) model of rewards or a good (but not necessarily consistent) model
of past policy. Extensive empirical comparison demonstrates that the doubly
robust estimation uniformly improves over existing techniques, achieving both
lower variance in value estimation and better policies. As such, we expect the
doubly robust approach to become common practice in policy evaluation and
optimization.
