  State-of-the-art models for high-resolution image generation, such as BigGAN
and VQVAE-2, require an incredible amount of compute resources and/or time (512
TPU-v3 cores) to train, putting them out of reach for the larger research
community. On the other hand, GAN-based image super-resolution models, such as
ESRGAN, can not only upscale images to high dimensions, but also are efficient
to train. In this paper, we present not-so-big-GAN (nsb-GAN), a simple yet
cost-effective two-step training framework for deep generative models (DGMs) of
high-dimensional natural images. First, we generate images in low-frequency
bands by training a sampler in the wavelet domain. Then, we super-resolve these
images from the wavelet domain back to the pixel-space with our novel wavelet
super-resolution decoder network. Wavelet-based down-sampling method preserves
more structural information than pixel-based methods, leading to significantly
better generative quality of the low-resolution sampler (e.g., 64x64). Since
the sampler and decoder can be trained in parallel and operate on much lower
dimensional spaces than end-to-end models, the training cost is substantially
reduced. On ImageNet 512x512, our model achieves a Fr\'echet Inception Distance
(FID) of 10.59 -- beating the baseline BigGAN model -- at half the compute (256
TPU-v3 cores).
