  Evaluating novel contextual bandit policies using logged data is crucial in
applications where exploration is costly, such as medicine. But it usually
relies on the assumption of no unobserved confounders, which is bound to fail
in practice. We study the question of policy evaluation when we instead have
proxies for the latent confounders and develop an importance weighting method
that avoids fitting a latent outcome regression model. We show that unlike the
unconfounded case no single set of weights can give unbiased evaluation for all
outcome models, yet we propose a new algorithm that can still provably
guarantee consistency by instead minimizing an adversarial balance objective.
We further develop tractable algorithms for optimizing this objective and
demonstrate empirically the power of our method when confounders are latent.
