  In customer service technical support, swiftly and accurately retrieving
relevant past issues is critical for efficiently resolving customer inquiries.
The conventional retrieval methods in retrieval-augmented generation (RAG) for
large language models (LLMs) treat a large corpus of past issue tracking
tickets as plain text, ignoring the crucial intra-issue structure and
inter-issue relations, which limits performance. We introduce a novel customer
service question-answering method that amalgamates RAG with a knowledge graph
(KG). Our method constructs a KG from historical issues for use in retrieval,
retaining the intra-issue structure and inter-issue relations. During the
question-answering phase, our method parses consumer queries and retrieves
related sub-graphs from the KG to generate answers. This integration of a KG
not only improves retrieval accuracy by preserving customer service structure
information but also enhances answering quality by mitigating the effects of
text segmentation. Empirical assessments on our benchmark datasets, utilizing
key retrieval (MRR, Recall@K, NDCG@K) and text generation (BLEU, ROUGE, METEOR)
metrics, reveal that our method outperforms the baseline by 77.6% in MRR and by
0.32 in BLEU. Our method has been deployed within LinkedIn's customer service
team for approximately six months and has reduced the median per-issue
resolution time by 28.6%.
