  Fully convolutional neural networks can process input of arbitrary size by
applying a combination of downsampling and pooling. However, we find that fully
convolutional image classifiers are not agnostic to the input size but rather
show significant differences in performance: presenting the same image at
different scales can result in different outcomes. A closer look reveals that
there is no simple relationship between input size and model performance (no
`bigger is better'), but that each each network has a preferred input size, for
which it shows best results. We investigate this phenomenon by applying
different methods, including spectral analysis of layer activations and probe
classifiers, showing that there are characteristic features depending on the
network architecture. From this we find that the size of discriminatory
features is critically influencing how the inference process is distributed
among the layers.
