  Clinical decision making requires counterfactual reasoning based on a factual
medical image and thus necessitates causal image synthesis. To this end, we
present a novel method for modeling the causality between demographic
variables, clinical indices and brain MR images for Alzheimer's Diseases.
Specifically, we leverage a structural causal model to depict the causality and
a styled generator to synthesize the image. Furthermore, as a crucial step to
reduce modeling complexity and make learning tractable, we propose the use of
low dimensional latent feature representation of a high-dimensional 3D image,
together with exogenous noise, to build causal relationship between the image
and non image variables. We experiment the proposed method based on 1586
subjects and 3683 3D images and synthesize counterfactual brain MR images
intervened on certain attributes, such as age, brain volume and cognitive test
score. Quantitative metrics and qualitative evaluation of counterfactual images
demonstrates the superiority of our generated images.
