  Recent work has shown that infusing layout features into language models
(LMs) improves processing of visually-rich documents such as scientific papers.
Layout-infused LMs are often evaluated on documents with familiar layout
features (e.g., papers from the same publisher), but in practice models
encounter documents with unfamiliar distributions of layout features, such as
new combinations of text sizes and styles, or new spatial configurations of
textual elements. In this work we test whether layout-infused LMs are robust to
layout distribution shifts. As a case study we use the task of scientific
document structure recovery, segmenting a scientific paper into its structural
categories (e.g., "title", "caption", "reference"). To emulate distribution
shifts that occur in practice we re-partition the GROTOAP2 dataset. We find
that under layout distribution shifts model performance degrades by up to 20
F1. Simple training strategies, such as increasing training diversity, can
reduce this degradation by over 35% relative F1; however, models fail to reach
in-distribution performance in any tested out-of-distribution conditions. This
work highlights the need to consider layout distribution shifts during model
evaluation, and presents a methodology for conducting such evaluations.
