  Recent mask proposal models have significantly improved the performance of
zero-shot semantic segmentation. However, the use of a `background' embedding
during training in these methods is problematic as the resulting model tends to
over-learn and assign all unseen classes as the background class instead of
their correct labels. Furthermore, they ignore the semantic relationship of
text embeddings, which arguably can be highly informative for zero-shot
prediction as seen classes may have close relationship with unseen classes. To
this end, this paper proposes novel class enhancement losses to bypass the use
of the background embbedding during training, and simultaneously exploit the
semantic relationship between text embeddings and mask proposals by ranking the
similarity scores. To further capture the relationship between seen and unseen
classes, we propose an effective pseudo label generation pipeline using
pretrained vision-language model. Extensive experiments on several benchmark
datasets show that our method achieves overall the best performance for
zero-shot semantic segmentation. Our method is flexible, and can also be
applied to the challenging open-vocabulary semantic segmentation problem.
