  Existing evaluations of tool learning primarily focus on validating the
alignment of selected tools for large language models (LLMs) with expected
outcomes. However, these approaches rely on a limited set of scenarios where
answers can be pre-determined, diverging from genuine needs. Furthermore, a
sole emphasis on outcomes disregards the intricate capabilities essential for
LLMs to effectively utilize tools. To tackle this issue, we propose ToolEyes, a
fine-grained system tailored for the evaluation of the LLMs' tool learning
capabilities in authentic scenarios. The system meticulously examines seven
real-world scenarios, analyzing five dimensions crucial to LLMs in tool
learning: format alignment, intent comprehension, behavior planning, tool
selection, and answer organization. Additionally, ToolEyes incorporates a tool
library boasting approximately 600 tools, serving as an intermediary between
LLMs and the physical world. Evaluations involving ten LLMs across three
categories reveal a preference for specific scenarios and limited cognitive
abilities in tool learning. Intriguingly, expanding the model size even
exacerbates the hindrance to tool learning. These findings offer instructive
insights aimed at advancing the field of tool learning. The data is available
att https://github.com/Junjie-Ye/ToolEyes.
