  Removing rain streaks from rainy images is necessary for many tasks in
computer vision, such as object detection and recognition. It needs to address
two mutually exclusive objectives: removing rain streaks and reserving
realistic details. Balancing them is critical for de-raining methods. We
propose an end-to-end network, called dual-task de-raining network (DTDN),
consisting of two sub-networks: generative adversarial network (GAN) and
convolutional neural network (CNN), to remove rain streaks via coordinating the
two mutually exclusive objectives self-adaptively. DTDN-GAN is mainly used to
remove structural rain streaks, and DTDN-CNN is designed to recover details in
original images. We also design a training algorithm to train these two
sub-networks of DTDN alternatively, which share same weights but use different
training sets. We further enrich two existing datasets to approximate the
distribution of real rain streaks. Experimental results show that our method
outperforms several recent state-of-the-art methods, based on both benchmark
testing datasets and real rainy images.
