  Retrieval augmented generation (RAG) systems combine the strengths of
language generation and information retrieval to power many real-world
applications like chatbots. Use of RAG for combined understanding of multimodal
data such as text, images and videos is appealing but two critical limitations
exist: one-time, upfront capture of all content in large multimodal data as
text descriptions entails high processing times, and not all information in the
rich multimodal data is typically in the text descriptions. Since the user
queries are not known apriori, developing a system for multimodal to text
conversion and interactive querying of multimodal data is challenging.
  To address these limitations, we propose iRAG, which augments RAG with a
novel incremental workflow to enable interactive querying of large corpus of
multimodal data. Unlike traditional RAG, iRAG quickly indexes large
repositories of multimodal data, and in the incremental workflow, it uses the
index to opportunistically extract more details from select portions of the
multimodal data to retrieve context relevant to an interactive user query. Such
an incremental workflow avoids long multimodal to text conversion times,
overcomes information loss issues by doing on-demand query-specific extraction
of details in multimodal data, and ensures high quality of responses to
interactive user queries that are often not known apriori. To the best of our
knowledge, iRAG is the first system to augment RAG with an incremental workflow
to support efficient interactive querying of large, real-world multimodal data.
Experimental results on real-world long videos demonstrate 23x to 25x faster
video to text ingestion, while ensuring that quality of responses to
interactive user queries is comparable to responses from a traditional RAG
where all video data is converted to text upfront before any querying.
