  Software engineers are increasingly adding semantic search capabilities to
applications using a strategy known as Retrieval Augmented Generation (RAG). A
RAG system involves finding documents that semantically match a query and then
passing the documents to a large language model (LLM) such as ChatGPT to
extract the right answer using an LLM. RAG systems aim to: a) reduce the
problem of hallucinated responses from LLMs, b) link sources/references to
generated responses, and c) remove the need for annotating documents with
meta-data. However, RAG systems suffer from limitations inherent to information
retrieval systems and from reliance on LLMs. In this paper, we present an
experience report on the failure points of RAG systems from three case studies
from separate domains: research, education, and biomedical. We share the
lessons learned and present 7 failure points to consider when designing a RAG
system. The two key takeaways arising from our work are: 1) validation of a RAG
system is only feasible during operation, and 2) the robustness of a RAG system
evolves rather than designed in at the start. We conclude with a list of
potential research directions on RAG systems for the software engineering
community.
