  Tumor segmentation in multimodal medical images has seen a growing trend
towards deep learning based methods. Typically, studies dealing with this topic
fuse multimodal image data to improve the tumor segmentation contour for a
single imaging modality. However, they do not take into account that tumor
characteristics are emphasized differently by each modality, which affects the
tumor delineation. Thus, the tumor segmentation is modality- and
task-dependent. This is especially the case for soft tissue sarcomas, where,
due to necrotic tumor tissue, the segmentation differs vastly. Closing this
gap, we develop a modalityspecific sarcoma segmentation model that utilizes
multimodal image data to improve the tumor delineation on each individual
modality. We propose a simultaneous co-segmentation method, which enables
multimodal feature learning through modality-specific encoder and decoder
branches, and the use of resource-effcient densely connected convolutional
layers. We further conduct experiments to analyze how different input
modalities and encoder-decoder fusion strategies affect the segmentation
result. We demonstrate the effectiveness of our approach on public soft tissue
sarcoma data, which comprises MRI (T1 and T2 sequence) and PET/CT scans. The
results show that our multimodal co-segmentation model provides better
modality-specific tumor segmentation than models using only the PET or MRI (T1
and T2) scan as input.
