  The use of Retrieval-Augmented Generation (RAG) has improved Large Language
Models (LLMs) in collaborating with external data, yet significant challenges
exist in real-world scenarios. In areas such as academic literature and finance
question answering, data are often found in raw text and tables in HTML or PDF
formats, which can be lengthy and highly unstructured. In this paper, we
introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that
involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We
revisit popular LLM- and RAG-based solutions for document analysis and evaluate
the design choices and answer qualities across multiple document domains and
diverse query types. Our evaluation yields interesting findings and highlights
the importance of data parsing and retrieval. We hope our benchmark can shed
light and better serve real-world document analysis applications. The benchmark
suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.
