  Integrating multi-modal data to promote medical image analysis has recently
gained great attention. This paper presents a novel scheme to learn the mutual
benefits of different modalities to achieve better segmentation results for
unpaired multi-modal medical images. Our approach tackles two critical issues
of this task from a practical perspective: (1) how to effectively learn the
semantic consistencies of various modalities (e.g., CT and MRI), and (2) how to
leverage the above consistencies to regularize the network learning while
preserving its simplicity. To address (1), we leverage a carefully designed
External Attention Module (EAM) to align semantic class representations and
their correlations of different modalities. To solve (2), the proposed EAM is
designed as an external plug-and-play one, which can be discarded once the
model is optimized. We have demonstrated the effectiveness of the proposed
method on two medical image segmentation scenarios: (1) cardiac structure
segmentation, and (2) abdominal multi-organ segmentation. Extensive results
show that the proposed method outperforms its counterparts by a wide margin.
