  Semantic segmentation is an important task for scene understanding in
self-driving cars and robotics, which aims to assign dense labels for all
pixels in the image. Existing work typically improves semantic segmentation
performance by exploring different network architectures on a target dataset.
Little attention has been paid to build a unified system by simultaneously
learning from multiple datasets due to the inherent distribution shift across
different datasets. In this paper, we propose a simple, flexible, and general
method for semantic segmentation, termed Cross-Dataset Collaborative Learning
(CDCL). Our goal is to train a unified model for improving the performance in
each dataset by leveraging information from all the datasets. Specifically, we
first introduce a family of Dataset-Aware Blocks (DAB) as the fundamental
computing units of the network, which help capture homogeneous convolutional
representations and heterogeneous statistics across different datasets. Second,
we present a Dataset Alternation Training (DAT) mechanism to facilitate the
collaborative optimization procedure. We conduct extensive evaluations on
diverse semantic segmentation datasets for autonomous driving. Experiments
demonstrate that our method consistently achieves notable improvements over
prior single-dataset and cross-dataset training methods without introducing
extra FLOPs. Particularly, with the same architecture of PSPNet (ResNet-18),
our method outperforms the single-dataset baseline by 5.65\%, 6.57\%, and
5.79\% mIoU on the validation sets of Cityscapes, BDD100K, CamVid,
respectively. We also apply CDCL for point cloud 3D semantic segmentation and
achieve improved performance, which further validates the superiority and
generality of our method. Code and models will be released.
