  Conditional Generative Adversarial Networks (cGANs) extend the standard
unconditional GAN framework to learning joint data-label distributions from
samples, and have been established as powerful generative models capable of
generating high-fidelity imagery. A challenge of training such a model lies in
properly infusing class information into its generator and discriminator. For
the discriminator, class conditioning can be achieved by either (1) directly
incorporating labels as input or (2) involving labels in an auxiliary
classification loss. In this paper, we show that the former directly aligns the
class-conditioned fake-and-real data distributions
$P(\text{image}|\text{class})$ ({\em data matching}), while the latter aligns
data-conditioned class distributions $P(\text{class}|\text{image})$ ({\em label
matching}). Although class separability does not directly translate to sample
quality and becomes a burden if classification itself is intrinsically
difficult, the discriminator cannot provide useful guidance for the generator
if features of distinct classes are mapped to the same point and thus become
inseparable. Motivated by this intuition, we propose a Dual Projection GAN
(P2GAN) model that learns to balance between {\em data matching} and {\em label
matching}. We then propose an improved cGAN model with Auxiliary Classification
that directly aligns the fake and real conditionals
$P(\text{class}|\text{image})$ by minimizing their $f$-divergence. Experiments
on a synthetic Mixture of Gaussian (MoG) dataset and a variety of real-world
datasets including CIFAR100, ImageNet, and VGGFace2 demonstrate the efficacy of
our proposed models.
