  Measuring the naturalness of images is important to generate realistic images
or to detect unnatural regions in images. Additionally, a method to measure
naturalness can be complementary to Convolutional Neural Network (CNN) based
features, which are known to be insensitive to the naturalness of images.
However, most probabilistic image models have insufficient capability of
modeling the complex and abstract naturalness that we feel because they are
built directly on raw image pixels. In this work, we assume that naturalness
can be measured by the predictability on high-level features during eye
movement. Based on this assumption, we propose a novel method to evaluate the
naturalness by building a variant of Recurrent Neural Network Language Models
on pre-trained CNN representations. Our method is applied to two tasks,
demonstrating that 1) using our method as a regularizer enables us to generate
more understandable images from image features than existing approaches, and 2)
unnaturalness maps produced by our method achieve state-of-the-art eye fixation
prediction performance on two well-studied datasets.
