  Instruction-tuned large language models (LLMs) excel at many tasks but often
fail to use external tools due to complicated and unfamiliar syntax
constraints. While extensive fine-tuning and prompting can mitigate the issue,
these approaches are expensive and hard to generalize. Furthermore, because
syntax constraints are only learned implicitly during fine-tuning, models still
make frequent syntax errors. Motivated by the fact that these constraints can
be better satisfied explicitly with constrained decoding, we propose TOOLDEC, a
decoding algorithm using finite state machines to force LLMs to follow tool
syntax. Our experiments show that TOOLDEC eliminates all syntax errors,
achieving significantly better performance on various base models and
benchmarks. More surprisingly, when applied to generalist out-of-the-box LLMs
such as Mistral-Instruct, TOOLDEC improves its accuracy in tool use from the
initial 0% to an impressive 52%, matching the performance of specialized
fine-tuned models such as ToolLLM.
