  Maintaining natural image statistics is a crucial factor in restoration and
generation of realistic looking images. When training CNNs, photorealism is
usually attempted by adversarial training (GAN), that pushes the output images
to lie on the manifold of natural images. GANs are very powerful, but not
perfect. They are hard to train and the results still often suffer from
artifacts. In this paper we propose a complementary approach, that could be
applied with or without GAN, whose goal is to train a feed-forward CNN to
maintain natural internal statistics. We look explicitly at the distribution of
features in an image and train the network to generate images with natural
feature distributions. Our approach reduces by orders of magnitude the number
of images required for training and achieves state-of-the-art results on both
single-image super-resolution, and high-resolution surface normal estimation.
