  Conformal prediction is a distribution-free method that wraps a given machine
learning model and returns a set of plausible labels that contain the true
label with a prescribed coverage rate. In practice, the empirical coverage
achieved highly relies on fully observed label information from data both in
the training phase for model fitting and the calibration phase for quantile
estimation. This dependency poses a challenge in the context of online learning
with bandit feedback, where a learner only has access to the correctness of
actions (i.e., pulled an arm) but not the full information of the true label.
In particular, when the pulled arm is incorrect, the learner only knows that
the pulled one is not the true class label, but does not know which label is
true. Additionally, bandit feedback further results in a smaller labeled
dataset for calibration, limited to instances with correct actions, thereby
affecting the accuracy of quantile estimation. To address these limitations, we
propose Bandit Class-specific Conformal Prediction (BCCP), offering coverage
guarantees on a class-specific granularity. Using an unbiased estimation of an
estimand involving the true label, BCCP trains the model and makes set-valued
inferences through stochastic gradient descent. Our approach overcomes the
challenges of sparsely labeled data in each iteration and generalizes the
reliability and applicability of conformal prediction to online decision-making
environments.
