  Convolutional Neural Networks (CNNs) are state-of-the-art models for document
image classification tasks. However, many of these approaches rely on
parameters and architectures designed for classifying natural images, which
differ from document images. We question whether this is appropriate and
conduct a large empirical study to find what aspects of CNNs most affect
performance on document images. Among other results, we exceed the
state-of-the-art on the RVL-CDIP dataset by using shear transform data
augmentation and an architecture designed for a larger input image.
Additionally, we analyze the learned features and find evidence that CNNs
trained on RVL-CDIP learn region-specific layout features.
