  Deploying deep neural networks for risk-sensitive tasks necessitates an
uncertainty estimation mechanism. This paper introduces hierarchical selective
classification, extending selective classification to a hierarchical setting.
Our approach leverages the inherent structure of class relationships, enabling
models to reduce the specificity of their predictions when faced with
uncertainty. In this paper, we first formalize hierarchical risk and coverage,
and introduce hierarchical risk-coverage curves. Next, we develop algorithms
for hierarchical selective classification (which we refer to as "inference
rules"), and propose an efficient algorithm that guarantees a target accuracy
constraint with high probability. Lastly, we conduct extensive empirical
studies on over a thousand ImageNet classifiers, revealing that training
regimes such as CLIP, pretraining on ImageNet21k and knowledge distillation
boost hierarchical selective performance.
