  We propose a compact and effective framework to fuse multimodal features at
multiple layers in a single network. The framework consists of two innovative
fusion schemes. Firstly, unlike existing multimodal methods that necessitate
individual encoders for different modalities, we verify that multimodal
features can be learnt within a shared single network by merely maintaining
modality-specific batch normalization layers in the encoder, which also enables
implicit fusion via joint feature representation learning. Secondly, we propose
a bidirectional multi-layer fusion scheme, where multimodal features can be
exploited progressively. To take advantage of such scheme, we introduce two
asymmetric fusion operations including channel shuffle and pixel shift, which
learn different fused features with respect to different fusion directions.
These two operations are parameter-free and strengthen the multimodal feature
interactions across channels as well as enhance the spatial feature
discrimination within channels. We conduct extensive experiments on semantic
segmentation and image translation tasks, based on three publicly available
datasets covering diverse modalities. Results indicate that our proposed
framework is general, compact and is superior to state-of-the-art fusion
frameworks.
