  Visual-language foundation models, like CLIP, learn generalized
representations that enable zero-shot open-set classification. Few-shot
adaptation methods, based on prompt tuning, have been shown to further improve
performance on downstream datasets. However, these methods do not fare well in
the taxonomic open set (TOS) setting, where the classifier is asked to make
predictions from label sets across different levels of semantic granularity.
Frequently, they infer incorrect labels at coarser taxonomic class levels, even
when the inference at the leaf level (original class labels) is correct. To
address this problem, we propose a prompt tuning technique that calibrates the
hierarchical consistency of model predictions. A set of metrics of hierarchical
consistency, the Hierarchical Consistent Accuracy (HCA) and the Mean Treecut
Accuracy (MTA), are first proposed to evaluate TOS model performance. A new
Prompt Tuning for Hierarchical Consistency (ProTeCt) technique is then proposed
to calibrate classification across label set granularities. Results show that
ProTeCt can be combined with existing prompt tuning methods to significantly
improve TOS classification without degrading the leaf level classification
performance.
