  Answering questions within business and finance requires reasoning,
precision, and a wide-breadth of technical knowledge. Together, these
requirements make this domain difficult for large language models (LLMs). We
introduce BizBench, a benchmark for evaluating models' ability to reason about
realistic financial problems. BizBench comprises eight quantitative reasoning
tasks, focusing on question-answering (QA) over financial data via program
synthesis. We include three financially-themed code-generation tasks from newly
collected and augmented QA data. Additionally, we isolate the reasoning
capabilities required for financial QA: reading comprehension of financial text
and tables for extracting intermediate values, and understanding financial
concepts and formulas needed to calculate complex solutions. Collectively,
these tasks evaluate a model's financial background knowledge, ability to parse
financial documents, and capacity to solve problems with code. We conduct an
in-depth evaluation of open-source and commercial LLMs, comparing and
contrasting the behavior of code-focused and language-focused models. We
demonstrate that the current bottleneck in performance is due to LLMs' limited
business and financial understanding, highlighting the value of a challenging
benchmark for quantitative reasoning within this domain.
