  Given unstructured text, Large Language Models (LLMs) are adept at answering
simple (single-hop) questions. However, as the complexity of the questions
increase, the performance of LLMs degrade. We believe this is due to the
overhead associated with understanding the complex question followed by
filtering and aggregating unstructured information in the raw text. Recent
methods try to reduce this burden by integrating structured knowledge triples
into the raw text, aiming to provide a structured overview that simplifies
information processing. However, this simplistic approach is query-agnostic and
the extracted facts are ambiguous as they lack context. To address these
drawbacks and to enable LLMs to answer complex (multi-hop) questions with ease,
we propose to use a knowledge graph (KG) that is context-aware and is distilled
to contain query-relevant information. The use of our compressed distilled KG
as input to the LLM results in our method utilizing up to $67\%$ fewer tokens
to represent the query relevant information present in the supporting
documents, compared to the state-of-the-art (SoTA) method. Our experiments show
consistent improvements over the SoTA across several metrics (EM, F1,
BERTScore, and Human Eval) on two popular benchmark datasets (HotpotQA and
MuSiQue).
