  Recent works like Tree-of-Thought (ToT) and Reasoning via Planning (RAP) aim
to augment the reasoning capabilities of LLMs by using tree-search algorithms
to guide multi-step reasoning. These methods rely on prompting a pre-trained
model to serve as a value function and focus on problems with low search depth.
As a result, these methods will not work in domains where the pre-trained LLM
does not have enough knowledge to serve as an effective value function or in
domains that require long-horizon planning. To address these limitations, we
present an AlphaZero-like tree-search learning framework for LLMs (termed
TS-LLM), systematically illustrating how tree-search with a learned value
function can guide LLM decoding. TS-LLM distinguishes itself in two key ways.
(1) Leveraging a learned value function and AlphaZero-like algorithms, our
approach can be generally adaptable to a wide range of tasks, language models
of any size, and tasks of varying search depths. (2) Our approach can guide
LLMs during both inference and training, iteratively improving the LLM.
Empirical results across reasoning, planning, alignment, and decision-making
tasks show that TS-LLM outperforms existing approaches and can handle trees
with a depth of 64.
