  Classification of document images is a critical step for archival of old
manuscripts, online subscription and administrative procedures. Computer vision
and deep learning have been suggested as a first solution to classify documents
based on their visual appearance. However, achieving the fine-grained
classification that is required in real-world setting cannot be achieved by
visual analysis alone. Often, the relevant information is in the actual text
content of the document. We design a multimodal neural network that is able to
learn from word embeddings, computed on text extracted by OCR, and from the
image. We show that this approach boosts pure image accuracy by 3% on
Tobacco3482 and RVL-CDIP augmented by our new QS-OCR text dataset
(https://github.com/Quicksign/ocrized-text-dataset), even without clean text
information.
