  Semantic segmentation is important for many real-world systems, e.g.,
autonomous vehicles, which predict the class of each pixel. Recently, deep
networks achieved significant progress w.r.t. the mean Intersection-over Union
(mIoU) with the cross-entropy loss. However, the cross-entropy loss can
essentially ignore the difference of severity for an autonomous car with
different wrong prediction mistakes. For example, predicting the car to the
road is much more servery than recognize it as the bus. Targeting for this
difficulty, we develop a Wasserstein training framework to explore the
inter-class correlation by defining its ground metric as misclassification
severity. The ground metric of Wasserstein distance can be pre-defined
following the experience on a specific task. From the optimization perspective,
we further propose to set the ground metric as an increasing function of the
pre-defined ground metric. Furthermore, an adaptively learning scheme of the
ground matrix is proposed to utilize the high-fidelity CARLA simulator.
Specifically, we follow a reinforcement alternative learning scheme. The
experiments on both CamVid and Cityscapes datasets evidenced the effectiveness
of our Wasserstein loss. The SegNet, ENet, FCN and Deeplab networks can be
adapted following a plug-in manner. We achieve significant improvements on the
predefined important classes, and much longer continuous playtime in our
simulator.
