  We present compositional nearest neighbors (CompNN), a simple approach to
visually interpreting distributed representations learned by a convolutional
neural network (CNN) for pixel-level tasks (e.g., image synthesis and
segmentation). It does so by reconstructing both a CNN's input and output image
by copy-pasting corresponding patches from the training set with similar
feature embeddings. To do so efficiently, it makes of a patch-match-based
algorithm that exploits the fact that the patch representations learned by a
CNN for pixel level tasks vary smoothly. Finally, we show that CompNN can be
used to establish semantic correspondences between two images and control
properties of the output image by modifying the images contained in the
training set. We present qualitative and quantitative experiments for semantic
segmentation and image-to-image translation that demonstrate that CompNN is a
good tool for interpreting the embeddings learned by pixel-level CNNs.
