  Many real-world problems require the combined application of multiple
reasoning abilities employing suitable abstractions, commonsense knowledge, and
creative synthesis of problem-solving strategies. To help advance AI systems
towards such capabilities, we propose a new reasoning challenge, namely Fermi
Problems (FPs), which are questions whose answers can only be approximately
estimated because their precise computation is either impractical or
impossible. For example, "How much would the sea level rise if all ice in the
world melted?" FPs are commonly used in quizzes and interviews to bring out and
evaluate the creative reasoning abilities of humans. To do the same for AI
systems, we present two datasets: 1) A collection of 1k real-world FPs sourced
from quizzes and olympiads; and 2) a bank of 10k synthetic FPs of intermediate
complexity to serve as a sandbox for the harder real-world challenge. In
addition to question answer pairs, the datasets contain detailed solutions in
the form of an executable program and supporting facts, helping in supervision
and evaluation of intermediate steps. We demonstrate that even extensively
fine-tuned large scale language models perform poorly on these datasets, on
average making estimates that are off by two orders of magnitude. Our
contribution is thus the crystallization of several unsolved AI problems into a
single, new challenge that we hope will spur further advances in building
systems that can reason.
