  We consider the contextual bandit problem where at each time, the agent only
has access to a noisy version of the context and the error variance (or an
estimator of this variance). This setting is motivated by a wide range of
applications where the true context for decision-making is unobserved, and only
a prediction of the context by a potentially complex machine learning algorithm
is available. When the context error is non-vanishing, classical bandit
algorithms fail to achieve sublinear regret. We propose the first online
algorithm in this setting with sublinear regret guarantees under mild
conditions. The key idea is to extend the measurement error model in classical
statistics to the online decision-making setting, which is nontrivial due to
the policy being dependent on the noisy context observations. We further
demonstrate the benefits of the proposed approach in simulation environments
based on synthetic and real digital intervention datasets.
