  Retrieval-augmented generation (RAG) utilizes retrieved texts to enhance
large language models (LLMs). However, studies show that RAG is not
consistently effective and can even mislead LLMs due to noisy or incorrect
retrieved texts. This suggests that RAG possesses a duality including both
benefit and detriment. Although many existing methods attempt to address this
issue, they lack a theoretical explanation for the duality in RAG. The benefit
and detriment within this duality remain a black box that cannot be quantified
or compared in an explainable manner. This paper takes the first step in
theoretically giving the essential explanation of benefit and detriment in RAG
by: (1) decoupling and formalizing them from RAG prediction, (2) approximating
the gap between their values by representation similarity and (3) establishing
the trade-off mechanism between them, to make them explainable, quantifiable,
and comparable. We demonstrate that the distribution difference between
retrieved texts and LLMs' knowledge acts as double-edged sword, bringing both
benefit and detriment. We also prove that the actual effect of RAG can be
predicted at token level. Based on our theory, we propose a practical novel
method, X-RAG, which achieves collaborative generation between pure LLM and RAG
at token level to preserve benefit and avoid detriment. Experiments in
real-world tasks based on LLMs including OPT, LLaMA-2, and Mistral show the
effectiveness of our method and support our theoretical results.
