  This paper highlights the need to bring document classification benchmarking
closer to real-world applications, both in the nature of data tested ($X$:
multi-channel, multi-paged, multi-industry; $Y$: class distributions and label
set variety) and in classification tasks considered ($f$: multi-page document,
page stream, and document bundle classification, ...). We identify the lack of
public multi-page document classification datasets, formalize different
classification tasks arising in application scenarios, and motivate the value
of targeting efficient multi-page document representations. An experimental
study on proposed multi-page document classification datasets demonstrates that
current benchmarks have become irrelevant and need to be updated to evaluate
complete documents, as they naturally occur in practice. This reality check
also calls for more mature evaluation methodologies, covering calibration
evaluation, inference complexity (time-memory), and a range of realistic
distribution shifts (e.g., born-digital vs. scanning noise, shifting page
order). Our study ends on a hopeful note by recommending concrete avenues for
future improvements.}
