  Unpaired medical image synthesis aims to provide complementary information
for an accurate clinical diagnostics, and address challenges in obtaining
aligned multi-modal medical scans. Transformer-based models excel in imaging
translation tasks thanks to their ability to capture long-range dependencies.
Although effective in supervised training settings, their performance falters
in unpaired image synthesis, particularly in synthesizing structural details.
This paper empirically demonstrates that, lacking strong inductive biases,
Transformer can converge to non-optimal solutions in the absence of paired
data. To address this, we introduce UNet Structured Transformer (UNest), a
novel architecture incorporating structural inductive biases for unpaired
medical image synthesis. We leverage the foundational Segment-Anything Model to
precisely extract the foreground structure and perform structural attention
within the main anatomy. This guides the model to learn key anatomical regions,
thus improving structural synthesis under the lack of supervision in unpaired
training. Evaluated on two public datasets, spanning three modalities, i.e.,
MR, CT, and PET, UNest improves recent methods by up to 19.30% across six
medical image synthesis tasks. Our code is released at
https://github.com/HieuPhan33/MICCAI2024-UNest.
