  Deep neural networks suffer from over-fitting and catastrophic forgetting
when trained with small data. One natural remedy for this problem is data
augmentation, which has been recently shown to be effective. However, previous
works either assume that intra-class variances can always be generalized to new
classes, or employ naive generation methods to hallucinate finite examples
without modeling their latent distributions. In this work, we propose
Covariance-Preserving Adversarial Augmentation Networks to overcome existing
limits of low-shot learning. Specifically, a novel Generative Adversarial
Network is designed to model the latent distribution of each novel class given
its related base counterparts. Since direct estimation of novel classes can be
inductively biased, we explicitly preserve covariance information as the
`variability' of base examples during the generation process. Empirical results
show that our model can generate realistic yet diverse examples, leading to
substantial improvements on the ImageNet benchmark over the state of the art.
