  The recent trend of using Large Language Models (LLMs) as tool agents in
real-world applications underscores the necessity for comprehensive evaluations
of their capabilities, particularly in complex scenarios involving planning,
creating, and using tools. However, existing benchmarks typically focus on
simple synthesized queries that do not reflect real-world complexity, thereby
offering limited perspectives in evaluating tool utilization. To address this
issue, we present UltraTool, a novel benchmark designed to improve and evaluate
LLMs' ability in tool utilization within real-world scenarios. UltraTool
focuses on the entire process of using tools - from planning and creating to
applying them in complex tasks. It emphasizes real-world complexities,
demanding accurate, multi-step planning for effective problem-solving. A key
feature of UltraTool is its independent evaluation of planning with natural
language, which happens before tool usage and simplifies the task solving by
mapping out the intermediate steps. Thus, unlike previous work, it eliminates
the restriction of pre-defined toolset. Through extensive experiments on
various LLMs, we offer novel insights into the evaluation of capabilities of
LLMs in tool utilization, thereby contributing a fresh perspective to this
rapidly evolving field. The benchmark is publicly available at
https://github.com/JoeYing1019/UltraTool.
