  Scaling up language models to billions of parameters has opened up
possibilities for in-context learning, allowing instruction tuning and few-shot
learning on tasks that the model was not specifically trained for. This has
achieved breakthrough performance on language tasks such as translation,
summarization, and question-answering. Furthermore, in addition to these
associative "System 1" tasks, recent advances in Chain-of-thought prompt
learning have demonstrated strong "System 2" reasoning abilities, answering a
question in the field of artificial general intelligence whether LLMs can
reason. The field started with the question whether LLMs can solve grade school
math word problems. This paper reviews the rapidly expanding field of
prompt-based reasoning with LLMs. Our taxonomy identifies different ways to
generate, evaluate, and control multi-step reasoning. We provide an in-depth
coverage of core approaches and open problems, and we propose a research agenda
for the near future. Finally, we highlight the relation between reasoning and
prompt-based learning, and we discuss the relation between reasoning,
sequential decision processes, and reinforcement learning. We find that
self-improvement, self-reflection, and some metacognitive abilities of the
reasoning processes are possible through the judicious use of prompts. True
self-improvement and self-reasoning, to go from reasoning with LLMs to
reasoning by LLMs, remains future work.
