  We study the problem of conditional generative modeling based on designated
semantics or structures. Existing models that build conditional generators
either require massive labeled instances as supervision or are unable to
accurately control the semantics of generated samples. We propose structured
generative adversarial networks (SGANs) for semi-supervised conditional
generative modeling. SGAN assumes the data x is generated conditioned on two
independent latent variables: y that encodes the designated semantics, and z
that contains other factors of variation. To ensure disentangled semantics in y
and z, SGAN builds two collaborative games in the hidden space to minimize the
reconstruction error of y and z, respectively. Training SGAN also involves
solving two adversarial games that have their equilibrium concentrating at the
true joint data distributions p(x, z) and p(x, y), avoiding distributing the
probability mass diffusely over data space that MLE-based methods may suffer.
We assess SGAN by evaluating its trained networks, and its performance on
downstream tasks. We show that SGAN delivers a highly controllable generator,
and disentangled representations; it also establishes start-of-the-art results
across multiple datasets when applied for semi-supervised image classification
(1.27%, 5.73%, 17.26% error rates on MNIST, SVHN and CIFAR-10 using 50, 1000
and 4000 labels, respectively). Benefiting from the separate modeling of y and
z, SGAN can generate images with high visual quality and strictly following the
designated semantic, and can be extended to a wide spectrum of applications,
such as style transfer.
