  This work addresses the need for a balanced approach between performance and
efficiency in scalable production environments for visually-rich document
understanding (VDU) tasks. Currently, there is a reliance on large document
foundation models that offer advanced capabilities but come with a heavy
computational burden. In this paper, we propose a multimodal early exit (EE)
model design that incorporates various training strategies, exit layer types
and placements. Our goal is to achieve a Pareto-optimal balance between
predictive performance and efficiency for multimodal document image
classification. Through a comprehensive set of experiments, we compare our
approach with traditional exit policies and showcase an improved
performance-efficiency trade-off. Our multimodal EE design preserves the
model's predictive capabilities, enhancing both speed and latency. This is
achieved through a reduction of over 20% in latency, while fully retaining the
baseline accuracy. This research represents the first exploration of multimodal
EE design within the VDU community, highlighting as well the effectiveness of
calibration in improving confidence scores for exiting at different layers.
Overall, our findings contribute to practical VDU applications by enhancing
both performance and efficiency.
