  Multi-sensor clues have shown promise for object segmentation, but inherent
noise in each sensor, as well as the calibration error in practice, may bias
the segmentation accuracy. In this paper, we propose a novel approach by mining
the Cross-Modal Semantics to guide the fusion and decoding of multimodal
features, with the aim of controlling the modal contribution based on relative
entropy. We explore semantics among the multimodal inputs in two aspects: the
modality-shared consistency and the modality-specific variation. Specifically,
we propose a novel network, termed XMSNet, consisting of (1) all-round
attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer
self-supervision. On the one hand, the AF block explicitly dissociates the
shared and specific representation and learns to weight the modal contribution
by adjusting the \textit{proportion, region,} and \textit{pattern}, depending
upon the quality. On the other hand, our CFD initially decodes the shared
feature and then refines the output through specificity-aware querying.
Further, we enforce semantic consistency across the decoding layers to enable
interaction across network hierarchies, improving feature discriminability.
Exhaustive comparison on eleven datasets with depth or thermal clues, and on
two challenging tasks, namely salient and camouflage object segmentation,
validate our effectiveness in terms of both performance and robustness. The
source code is publicly available at https://github.com/Zongwei97/XMSNet.
