  Making online decisions can be challenging when features are sparse and
orthogonal to historical ones, especially when the optimal policy is learned
through collaborative filtering. We formulate the problem as a matrix
completion bandit (MCB), where the expected reward under each arm is
characterized by an unknown low-rank matrix. The $\epsilon$-greedy bandit and
the online gradient descent algorithm are explored. Policy learning and regret
performance are studied under a specific schedule for exploration probabilities
and step sizes. A faster decaying exploration probability yields smaller regret
but learns the optimal policy less accurately. We investigate an online
debiasing method based on inverse propensity weighting (IPW) and a general
framework for online policy inference. The IPW-based estimators are
asymptotically normal under mild arm-optimality conditions. Numerical
simulations corroborate our theoretical findings. Our methods are applied to
the San Francisco parking pricing project data, revealing intriguing
discoveries and outperforming the benchmark policy.
