  Most semantic segmentation models treat semantic segmentation as a pixel-wise
classification task and use a pixel-wise classification error as their
optimization criterions. However, the pixel-wise error ignores the strong
dependencies among the pixels in an image, which limits the performance of the
model. Several ways to incorporate the structure information of the objects
have been investigated, \eg, conditional random fields (CRF), image structure
priors based methods, and generative adversarial network (GAN). Nevertheless,
these methods usually require extra model branches or additional memories, and
some of them show limited improvements. In contrast, we propose a simple yet
effective structural similarity loss (SSL) to encode the structure information
of the objects, which only requires a few additional computational resources in
the training phase. Inspired by the widely-used structural similarity (SSIM)
index in image quality assessment, we use the linear correlation between two
images to quantify their structural similarity. And the goal of the proposed
SSL is to pay more attention to the positions, whose associated predictions
lead to a low degree of linear correlation between two corresponding regions in
the ground truth map and the predicted map. Thus the model can achieve a strong
structural similarity between the two maps through minimizing the SSL over the
whole map. The experimental results demonstrate that our method can achieve
substantial and consistent improvements in performance on the PASCAL VOC 2012
and Cityscapes datasets. The code will be released soon.
