  Diffusion models (DPMs) have demonstrated remarkable performance in image
generation, often times outperforming other generative models. Since their
introduction, the powerful noise-to-image denoising pipeline has been extended
to various discriminative tasks, including image segmentation. In case of
medical imaging, often times the images are large 3D scans, where segmenting
one image using DPMs become extremely inefficient due to large memory
consumption and time consuming iterative sampling process. In this work, we
propose a novel conditional generative modeling framework (LDSeg) that performs
diffusion in latent space for medical image segmentation. Our proposed
framework leverages the learned inherent low-dimensional latent distribution of
the target object shapes and source image embeddings. The conditional diffusion
in latent space not only ensures accurate n-D image segmentation for
multi-label objects, but also mitigates the major underlying problems of the
traditional DPM based segmentation: (1) large memory consumption, (2) time
consuming sampling process and (3) unnatural noise injection in forward/reverse
process. LDSeg achieved state-of-the-art segmentation accuracy on three medical
image datasets with different imaging modalities. Furthermore, we show that our
proposed model is significantly more robust to noises, compared to the
traditional deterministic segmentation models, which can be potential in
solving the domain shift problems in the medical imaging domain. Codes are
available at: https://github.com/LDSeg/LDSeg.
