  Leveraging external knowledge to enhance the reasoning ability is crucial for
commonsense question answering. However, the existing knowledge bases heavily
rely on manual annotation which unavoidably causes deficiency in coverage of
world-wide commonsense knowledge. Accordingly, the knowledge bases fail to be
flexible enough to support the reasoning over diverse questions. Recently,
large-scale language models (LLMs) have dramatically improved the intelligence
in capturing and leveraging knowledge, which opens up a new way to address the
issue of eliciting knowledge from language models. We propose a Unified Facts
Obtaining (UFO) approach. UFO turns LLMs into knowledge sources and produces
relevant facts (knowledge statements) for the given question. We first develop
a unified prompt consisting of demonstrations that cover different aspects of
commonsense and different question styles. On this basis, we instruct the LLMs
to generate question-related supporting facts for various commonsense questions
via prompting. After facts generation, we apply a dense retrieval-based fact
selection strategy to choose the best-matched fact. This kind of facts will be
fed into the answer inference model along with the question. Notably, due to
the design of unified prompts, UFO can support reasoning in various commonsense
aspects (including general commonsense, scientific commonsense, and social
commonsense). Extensive experiments on CommonsenseQA 2.0, OpenBookQA, QASC, and
Social IQA benchmarks show that UFO significantly improves the performance of
the inference model and outperforms manually constructed knowledge sources.
