  Large language models (LLMs) have shown superior performance without
task-specific fine-tuning. Despite the success, the knowledge stored in the
parameters of LLMs could still be incomplete and difficult to update due to the
computational costs. As complementary, retrieval-based methods can offer
non-parametric world knowledge and improve the performance on tasks such as
question answering. However, we find that the retrieved knowledge does not
always help and even has a negative impact on original responses occasionally.
To better make use of both internal knowledge and external world knowledge, we
investigate eliciting the model's ability to recognize what they know and do
not know (which is also called self-knowledge) and propose Self-Knowledge
guided Retrieval augmentation (SKR), a simple yet effective method which can
let LLMs refer to the questions they have previously encountered and adaptively
call for external resources when dealing with new questions. We evaluate SKR on
multiple datasets and demonstrate that it outperforms chain-of-thought based
and fully retrieval-based methods by using either InstructGPT or ChatGPT.
