  Integrating information from different reference data sources is a major
challenge for Retrieval-Augmented Generation (RAG) systems because each
knowledge source adopts a unique data structure and follows different
conventions. Retrieving from multiple knowledge sources with one fixed strategy
usually leads to under-exploitation of information. To mitigate this drawback,
inspired by Mix-of-Expert, we introduce Mix-of-Granularity (MoG), a method that
dynamically determines the optimal granularity of a knowledge database based on
input queries using a router. The router is efficiently trained with a newly
proposed loss function employing soft labels. We further extend MoG to
Mix-of-Granularity-Graph (MoGG), where reference documents are pre-processed
into graphs, enabling the retrieval of relevant information from distantly
situated chunks. Extensive experiments demonstrate that both MoG and MoGG
effectively predict optimal granularity levels, significantly enhancing the
performance of the RAG system in downstream tasks. The code of both MoG and
MoGG will be made public.
