  Open-vocabulary object detection (OvOD) has transformed detection into a
language-guided task, empowering users to freely define their class
vocabularies of interest during inference. However, our initial investigation
indicates that existing OvOD detectors exhibit significant variability when
dealing with vocabularies across various semantic granularities, posing a
concern for real-world deployment. To this end, we introduce Semantic Hierarchy
Nexus (SHiNe), a novel classifier that uses semantic knowledge from class
hierarchies. It runs offline in three steps: i) it retrieves relevant
super-/sub-categories from a hierarchy for each target class; ii) it integrates
these categories into hierarchy-aware sentences; iii) it fuses these sentence
embeddings to generate the nexus classifier vector. Our evaluation on various
detection benchmarks demonstrates that SHiNe enhances robustness across diverse
vocabulary granularities, achieving up to +31.9% mAP50 with ground truth
hierarchies, while retaining improvements using hierarchies generated by large
language models. Moreover, when applied to open-vocabulary classification on
ImageNet-1k, SHiNe improves the CLIP zero-shot baseline by +2.8% accuracy.
SHiNe is training-free and can be seamlessly integrated with any off-the-shelf
OvOD detector, without incurring additional computational overhead during
inference. The code is open source.
