  A typical desideratum for quantifying the uncertainty from a classification
model as a prediction set is class-conditional singleton set calibration. That
is, such sets should map to the output of well-calibrated selective
classifiers, matching the observed frequencies of similar instances. Recent
works proposing adaptive and localized conformal p-values for deep networks do
not guarantee this behavior, nor do they achieve it empirically. Instead, we
use the strong signals for prediction reliability from KNN-based approximations
of Transformer networks to construct data-driven partitions for Mondrian
Conformal Predictors, which are treated as weak selective classifiers that are
then calibrated via a new Inductive Venn Predictor, the Venn-ADMIT Predictor.
The resulting selective classifiers are well-calibrated, in a conservative but
practically useful sense for a given threshold. They are inherently robust to
changes in the proportions of the data partitions, and straightforward
conservative heuristics provide additional robustness to covariate shifts. We
compare and contrast to the quantities produced by recent Conformal Predictors
on several representative and challenging natural language processing
classification tasks, including class-imbalanced and distribution-shifted
settings.
