  Open-vocabulary semantic segmentation seeks to label each pixel in an image
with arbitrary text descriptions. Vision-language foundation models, especially
CLIP, have recently emerged as powerful tools for acquiring open-vocabulary
capabilities. However, fine-tuning CLIP to equip it with pixel-level prediction
ability often suffers three issues: 1) high computational cost, 2) misalignment
between the two inherent modalities of CLIP, and 3) degraded generalization
ability on unseen categories. To address these issues, we propose H-CLIP a
symmetrical parameter-efficient fine-tuning (PEFT) strategy conducted in
hyperspherical space for both of the two CLIP modalities. Specifically, the
PEFT strategy is achieved by a series of efficient block-diagonal learnable
transformation matrices and a dual cross-relation communication module among
all learnable matrices. Since the PEFT strategy is conducted symmetrically to
the two CLIP modalities, the misalignment between them is mitigated.
Furthermore, we apply an additional constraint to PEFT on the CLIP text encoder
according to the hyperspherical energy principle, i.e., minimizing
hyperspherical energy during fine-tuning preserves the intrinsic structure of
the original parameter space, to prevent the destruction of the generalization
ability offered by the CLIP text encoder. Extensive evaluations across various
benchmarks show that H-CLIP achieves new SOTA open-vocabulary semantic
segmentation results while only requiring updating approximately 4% of the
total parameters of CLIP.
