  Data analytics is essential for extracting valuable insights from data that
can assist organizations in making effective decisions. We introduce
InsightBench, a benchmark dataset with three key features. First, it consists
of 31 datasets representing diverse business use cases such as finance and
incident management, each accompanied by a carefully curated set of insights
planted in the datasets. Second, unlike existing benchmarks focusing on
answering single queries, InsightBench evaluates agents based on their ability
to perform end-to-end data analytics, including formulating questions,
interpreting answers, and generating a summary of insights and actionable
steps. Third, we conducted comprehensive quality assurance to ensure that each
dataset in the benchmark had clear goals and included relevant and meaningful
questions and analysis. Furthermore, we implement a two-way evaluation
mechanism using LLaMA-3-Eval as an effective, open-source evaluator method to
assess agents' ability to extract insights. We also propose AgentPoirot, our
baseline data analysis agent capable of performing end-to-end data analytics.
Our evaluation on InsightBench shows that AgentPoirot outperforms existing
approaches (such as Pandas Agent) that focus on resolving single queries. We
also compare the performance of open- and closed-source LLMs and various
evaluation strategies. Overall, this benchmark serves as a testbed to motivate
further development in comprehensive data analytics and can be accessed here:
https://github.com/ServiceNow/insight-bench.
