  The deep convolutional neural network has achieved significant progress for
single image rain streak removal. However, most of the data-driven learning
methods are full-supervised or semi-supervised, unexpectedly suffering from
significant performance drops when dealing with real rain. These data-driven
learning methods are representative yet generalize poor for real rain. The
opposite holds true for the model-driven unsupervised optimization methods. To
overcome these problems, we propose a unified unsupervised learning framework
which inherits the generalization and representation merits for real rain
removal. Specifically, we first discover a simple yet important domain
knowledge that directional rain streak is anisotropic while the natural clean
image is isotropic, and formulate the structural discrepancy into the energy
function of the optimization model. Consequently, we design an optimization
model-driven deep CNN in which the unsupervised loss function of the
optimization model is enforced on the proposed network for better
generalization. In addition, the architecture of the network mimics the main
role of the optimization models with better feature representation. On one
hand, we take advantage of the deep network to improve the representation. On
the other hand, we utilize the unsupervised loss of the optimization model for
better generalization. Overall, the unsupervised learning framework achieves
good generalization and representation: unsupervised training (loss) with only
a few real rainy images (input) and physical meaning network (architecture).
Extensive experiments on synthetic and real-world rain datasets show the
superiority of the proposed method.
