  Advances in the Visually-rich Document Understanding (VrDU) field and
particularly the Key-Information Extraction (KIE) task are marked with the
emergence of efficient Transformer-based approaches such as the LayoutLM
models. Despite the good performance of KIE models when fine-tuned on public
benchmarks, they still struggle to generalize on complex real-life use-cases
lacking sufficient document annotations. Our research highlighted that KIE
standard benchmarks such as SROIE and FUNSD contain significant similarity
between training and testing documents and can be adjusted to better evaluate
the generalization of models. In this work, we designed experiments to quantify
the information redundancy in public benchmarks, revealing a 75% template
replication in SROIE official test set and 16% in FUNSD. We also proposed
resampling strategies to provide benchmarks more representative of the
generalization ability of models. We showed that models not suited for document
analysis struggle on the adjusted splits dropping on average 10,5% F1 score on
SROIE and 3.5% on FUNSD compared to multi-modal models dropping only 7,5% F1 on
SROIE and 0.5% F1 on FUNSD.
