  We present RICHES, a novel approach that interleaves retrieval with sequence
generation tasks. RICHES offers an alternative to conventional RAG systems by
eliminating the need for separate retriever and generator. It retrieves
documents by directly decoding their contents, constrained on the corpus.
Unifying retrieval with generation allows us to adapt to diverse new tasks via
prompting alone. RICHES can work with any Instruction-tuned model, without
additional training. It provides attributed evidence, supports multi-hop
retrievals and interleaves thoughts to plan on what to retrieve next, all
within a single decoding pass of the LLM. We demonstrate the strong performance
of RICHES across ODQA tasks including attributed and multi-hop QA.
