  Deep neural networks (DNNs) have greatly contributed to the performance gains
in semantic segmentation. Nevertheless, training DNNs generally requires large
amounts of pixel-level labeled data, which is expensive and time-consuming to
collect in practice. To mitigate the annotation burden, this paper proposes a
self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain
data for semantic segmentation. In SE-GAN, a teacher network and a student
network constitute a self-ensembling model for generating semantic segmentation
maps, which together with a discriminator, forms a GAN. Despite its simplicity,
we find SE-GAN can significantly boost the performance of adversarial training
and enhance the stability of the model, the latter of which is a common barrier
shared by most adversarial training-based methods. We theoretically analyze
SE-GAN and provide an $\mathcal O(1/\sqrt{N})$ generalization bound ($N$ is the
training sample size), which suggests controlling the discriminator's
hypothesis complexity to enhance the generalizability. Accordingly, we choose a
simple network as the discriminator. Extensive and systematic experiments in
two standard settings demonstrate that the proposed method significantly
outperforms current state-of-the-art approaches. The source code of our model
is available online (https://github.com/YonghaoXu/SE-GAN).
