  Semantic segmentation requires per-pixel prediction for a given image.
Typically, the output resolution of a segmentation network is severely reduced
due to the downsampling operations in the CNN backbone. Most previous methods
employ upsampling decoders to recover the spatial resolution. Various decoders
were designed in the literature. Here, we propose a novel decoder, termed
dynamic neural representational decoder (NRD), which is simple yet
significantly more efficient. As each location on the encoder's output
corresponds to a local patch of the semantic labels, in this work, we represent
these local patches of labels with compact neural networks. This neural
representation enables our decoder to leverage the smoothness prior in the
semantic label space, and thus makes our decoder more efficient. Furthermore,
these neural representations are dynamically generated and conditioned on the
outputs of the encoder networks. The desired semantic labels can be efficiently
decoded from the neural representations, resulting in high-resolution semantic
segmentation predictions. We empirically show that our proposed decoder can
outperform the decoder in DeeplabV3+ with only 30% computational complexity,
and achieve competitive performance with the methods using dilated encoders
with only 15% computation. Experiments on the Cityscapes, ADE20K, and PASCAL
Context datasets demonstrate the effectiveness and efficiency of our proposed
method.
