  Due to the difficulty in collecting paired real-world training data, image
deraining is currently dominated by supervised learning with synthesized data
generated by e.g., Photoshop rendering. However, the generalization to real
rainy scenes is usually limited due to the gap between synthetic and real-world
data. In this paper, we first statistically explore why the supervised
deraining models cannot generalize well to real rainy cases, and find the
substantial difference of synthetic and real rainy data. Inspired by our
studies, we propose to remove rain by learning favorable deraining
representations from other connected tasks. In connected tasks, the label for
real data can be easily obtained. Hence, our core idea is to learn
representations from real data through task transfer to improve deraining
generalization. We thus term our learning strategy as \textit{task transfer
learning}. If there are more than one connected tasks, we propose to reduce
model size by knowledge distillation. The pretrained models for the connected
tasks are treated as teachers, all their knowledge is distilled to a student
network, so that we reduce the model size, meanwhile preserve effective prior
representations from all the connected tasks. At last, the student network is
fine-tuned with minority of paired synthetic rainy data to guide the pretrained
prior representations to remove rain. Extensive experiments demonstrate that
proposed task transfer learning strategy is surprisingly successful and
compares favorably with state-of-the-art supervised learning methods and
apparently surpass other semi-supervised deraining methods on synthetic data.
Particularly, it shows superior generalization over them to real-world scenes.
