  Document image classification is different from plain-text document
classification and consists of classifying a document by understanding the
content and structure of documents such as forms, emails, and other such
documents. We show that the only existing dataset for this task (Lewis et al.,
2006) has several limitations and we introduce two newly curated multilingual
datasets WIKI-DOC and MULTIEURLEX-DOC that overcome these limitations. We
further undertake a comprehensive study of popular visually-rich document
understanding or Document AI models in previously untested setting in document
image classification such as 1) multi-label classification, and 2) zero-shot
cross-lingual transfer setup. Experimental results show limitations of
multilingual Document AI models on cross-lingual transfer across typologically
distant languages. Our datasets and findings open the door for future research
into improving Document AI models.
