  Multimodal fusion and multitask learning are two vital topics in machine
learning. Despite the fruitful progress, existing methods for both problems are
still brittle to the same challenge -- it remains dilemmatic to integrate the
common information across modalities (resp. tasks) meanwhile preserving the
specific patterns of each modality (resp. task). Besides, while they are
actually closely related to each other, multimodal fusion and multitask
learning are rarely explored within the same methodological framework before.
In this paper, we propose Channel-Exchanging-Network (CEN) which is
self-adaptive, parameter-free, and more importantly, applicable for multimodal
and multitask dense image prediction. At its core, CEN adaptively exchanges
channels between subnetworks of different modalities. Specifically, the channel
exchanging process is self-guided by individual channel importance that is
measured by the magnitude of Batch-Normalization (BN) scaling factor during
training. For the application of dense image prediction, the validity of CEN is
tested by four different scenarios: multimodal fusion, cycle multimodal fusion,
multitask learning, and multimodal multitask learning. Extensive experiments on
semantic segmentation via RGB-D data and image translation through multi-domain
input verify the effectiveness of CEN compared to state-of-the-art methods.
Detailed ablation studies have also been carried out, which demonstrate the
advantage of each component we propose. Our code is available at
https://github.com/yikaiw/CEN.
