  Referring image segmentation aims at localizing all pixels of the visual
objects described by a natural language sentence. Previous works learn to
straightforwardly align the sentence embedding and pixel-level embedding for
highlighting the referred objects, but ignore the semantic consistency of
pixels within the same object, leading to incomplete masks and localization
errors in predictions. To tackle this problem, we propose CoupAlign, a simple
yet effective multi-level visual-semantic alignment method, to couple
sentence-mask alignment with word-pixel alignment to enforce object mask
constraint for achieving more accurate localization and segmentation.
Specifically, the Word-Pixel Alignment (WPA) module performs early fusion of
linguistic and pixel-level features in intermediate layers of the vision and
language encoders. Based on the word-pixel aligned embedding, a set of mask
proposals are generated to hypothesize possible objects. Then in the
Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence
embedding to localize the referred object, and finally projected back to
aggregate the pixels for the target. To further enhance the learning of the two
alignment modules, an auxiliary loss is designed to contrast the foreground and
background pixels. By hierarchically aligning pixels and masks with linguistic
features, our CoupAlign captures the pixel coherence at both visual and
semantic levels, thus generating more accurate predictions. Extensive
experiments on popular datasets (e.g., RefCOCO and G-Ref) show that our method
achieves consistent improvements over state-of-the-art methods, e.g., about 2%
oIoU increase on the validation and testing set of RefCOCO. Especially,
CoupAlign has remarkable ability in distinguishing the target from multiple
objects of the same class.
