  Acquiring annotated data at scale with rare diseases or conditions remains a
challenge. It would be extremely useful to have a method that controllably
synthesizes images that can correct such underrepresentation. Assuming a proper
latent representation, the idea of a "latent vector arithmetic" could offer the
means of achieving such synthesis. A proper representation must encode the
fidelity of the input data, preserve invariance and equivariance, and permit
arithmetic operations. Motivated by the ability to disentangle images into
spatial anatomy (tensor) factors and accompanying imaging (vector)
representations, we propose a framework termed "disentangled anatomy
arithmetic", in which a generative model learns to combine anatomical factors
of different input images such that when they are re-entangled with the desired
imaging modality (e.g. MRI), plausible new cardiac images are created with the
target characteristics. To encourage a realistic combination of anatomy factors
after the arithmetic step, we propose a localized noise injection network that
precedes the generator. Our model is used to generate realistic images,
pathology labels, and segmentation masks that are used to augment the existing
datasets and subsequently improve post-hoc classification and segmentation
tasks. Code is publicly available at https://github.com/vios-s/DAA-GAN.
