  We study contextual linear bandit problems under feature uncertainty; they
are noisy with missing entries. To address the challenges of the noise, we
analyze Bayesian oracles given observed noisy features. Our Bayesian analysis
finds that the optimal hypothesis can be far from the underlying realizability
function, depending on the noise characteristics, which are highly
non-intuitive and do not occur for classical noiseless setups. This implies
that classical approaches cannot guarantee a non-trivial regret bound.
Therefore, we propose an algorithm that aims at the Bayesian oracle from
observed information under this model, achieving $\tilde{O}(d\sqrt{T})$ regret
bound when there is a large number of arms. We demonstrate the proposed
algorithm using synthetic and real-world datasets.
