  Single image generation (SIG), described as generating diverse samples that
have similar visual content with the given single image, is first introduced by
SinGAN which builds a pyramid of GANs to progressively learn the internal patch
distribution of the single image. It also shows great potentials in a wide
range of image manipulation tasks. However, the paradigm of SinGAN has
limitations in terms of generation quality and training time. Firstly, due to
the lack of high-level information, SinGAN cannot handle the object images well
as it does on the scene and texture images. Secondly, the separate progressive
training scheme is time-consuming and easy to cause artifact accumulation. To
tackle these problems, in this paper, we dig into the SIG problem and improve
SinGAN by fully-utilization of internal and external priors. The main
contributions of this paper include: 1) We introduce to SIG a regularized
latent variable model. To the best of our knowledge, it is the first time to
give a clear formulation and optimization goal of SIG, and all the existing
methods for SIG can be regarded as special cases of this model. 2) We design a
novel Prior-based end-to-end training GAN (PetsGAN) to overcome the problems of
SinGAN. Our method gets rid of the time-consuming progressive training scheme
and can be trained end-to-end. 3) We construct abundant qualitative and
quantitative experiments to show the superiority of our method on both
generated image quality, diversity, and the training speed. Moreover, we apply
our method to other image manipulation tasks (e.g., style transfer,
harmonization), and the results further prove the effectiveness and efficiency
of our method.
