  We present the Cross-stitched Multi-task Unified Dual Recursive Network
(CMUDRN) model targeting the task of unified deraining and desnowing in a
multi-task learning setting. This unified model borrows from the basic Dual
Recursive Network (DRN) architecture developed by Cai et al. The proposed model
makes use of cross-stitch units that enable multi-task learning across two
separate DRN models, each tasked for single image deraining and desnowing,
respectively. By fixing cross-stitch units at several layers of basic
task-specific DRN networks, we perform multi-task learning over the two
separate DRN models. To enable blind image restoration, on top of these
structures we employ a simple neural fusion scheme which merges the output of
each DRN. The separate task-specific DRN models and the fusion scheme are
simultaneously trained by enforcing local and global supervision. Local
supervision is applied on the two DRN submodules, and global supervision is
applied on the data fusion submodule of the proposed model. Consequently, we
both enable feature sharing across task-specific DRN models and control the
image restoration behavior of the DRN submodules. An ablation study shows the
strength of the hypothesized CMUDRN model, and experiments indicate that its
performance is comparable or better than baseline DRN models on the single
image deraining and desnowing tasks. Moreover, CMUDRN enables blind image
restoration for the two underlying image restoration tasks, by unifying
task-specific image restoration pipelines via a naive parametric fusion scheme.
The CMUDRN implementation is available at https://github.com/VCL3D/CMUDRN.
