  Large Language Models (LLMs) are limited by their parametric knowledge,
leading to hallucinations in knowledge-extensive tasks. To address this,
Retrieval-Augmented Generation (RAG) incorporates external document chunks to
expand LLM knowledge. Furthermore, compressing information from document chunks
through extraction or summarization can improve LLM performance. Nonetheless,
LLMs still struggle to notice and utilize scattered key information, a problem
known as the "lost-in-the-middle" syndrome. Therefore, we typically need to
restructure the content for LLM to recognize the key information. We propose
$\textit{Refiner}$, an end-to-end extract-and-restructure paradigm that
operates in the post-retrieval process of RAG. $\textit{Refiner}$ leverages a
single decoder-only LLM to adaptively extract query-relevant contents verbatim
along with the necessary context, and section them based on their
interconnectedness, thereby highlights information distinction, and aligns
downstream LLMs with the original context effectively. Experiments show that a
trained $\textit{Refiner}$ (with 7B parameters) exhibits significant gain to
downstream LLM in improving answer accuracy, and outperforms other
state-of-the-art advanced RAG and concurrent compressing approaches in various
single-hop and multi-hop QA tasks. Notably, $\textit{Refiner}$ achieves a 80.5%
tokens reduction and a 1.6-7.0% improvement margin in multi-hop tasks compared
to the next best solution. $\textit{Refiner}$ is a plug-and-play solution that
can be seamlessly integrated with RAG systems, facilitating its application
across diverse open-source frameworks.
