  We propose an adversarial training procedure for learning a causal implicit
generative model for a given causal graph. We show that adversarial training
can be used to learn a generative model with true observational and
interventional distributions if the generator architecture is consistent with
the given causal graph. We consider the application of generating faces based
on given binary labels where the dependency structure between the labels is
preserved with a causal graph. This problem can be seen as learning a causal
implicit generative model for the image and labels. We devise a two-stage
procedure for this problem. First we train a causal implicit generative model
over binary labels using a neural network consistent with a causal graph as the
generator. We empirically show that WassersteinGAN can be used to output
discrete labels. Later, we propose two new conditional GAN architectures, which
we call CausalGAN and CausalBEGAN. We show that the optimal generator of the
CausalGAN, given the labels, samples from the image distributions conditioned
on these labels. The conditional GAN combined with a trained causal implicit
generative model for the labels is then a causal implicit generative model over
the labels and the generated image. We show that the proposed architectures can
be used to sample from observational and interventional image distributions,
even for interventions which do not naturally occur in the dataset.
