  The field of visually rich document understanding (VRDU) aims to solve a
multitude of well-researched NLP tasks in a multi-modal domain. Several
datasets exist for research on specific tasks of VRDU such as document
classification (DC), key entity extraction (KEE), entity linking, visual
question answering (VQA), inter alia. These datasets cover documents like
invoices and receipts with sparse annotations such that they support one or two
co-related tasks (e.g., entity extraction and entity linking). Unfortunately,
only focusing on a single specific of documents or task is not representative
of how documents often need to be processed in the wild - where variety in
style and requirements is expected. In this paper, we introduce BuDDIE
(Business Document Dataset for Information Extraction), the first multi-task
dataset of 1,665 real-world business documents that contains rich and dense
annotations for DC, KEE, and VQA. Our dataset consists of publicly available
business entity documents from US state government websites. The documents are
structured and vary in their style and layout across states and types (e.g.,
forms, certificates, reports, etc.). We provide data variety and quality
metrics for BuDDIE as well as a series of baselines for each task. Our
baselines cover traditional textual, multi-modal, and large language model
approaches to VRDU.
