  Large language models record impressive performance on many natural language
processing tasks. However, their knowledge capacity is limited to the
pretraining corpus. Retrieval augmentation offers an effective solution by
retrieving context from external knowledge sources to complement the language
model. However, existing retrieval augmentation techniques ignore the
structural relationships between these documents. Furthermore, retrieval models
are not explored much in scientific tasks, especially in regard to the
faithfulness of retrieved documents. In this paper, we propose a novel
structure-aware retrieval augmented language model that accommodates document
structure during retrieval augmentation. We create a heterogeneous document
graph capturing multiple types of relationships (e.g., citation, co-authorship,
etc.) that connect documents from more than 15 scientific disciplines (e.g.,
Physics, Medicine, Chemistry, etc.). We train a graph neural network on the
curated document graph to act as a structural encoder for the corresponding
passages retrieved during the model pretraining. Particularly, along with text
embeddings of the retrieved passages, we obtain structural embeddings of the
documents (passages) and fuse them together before feeding them to the language
model. We evaluate our model extensively on various scientific benchmarks that
include science question-answering and scientific document classification
tasks. Experimental results demonstrate that structure-aware retrieval improves
retrieving more coherent, faithful and contextually relevant passages, while
showing a comparable performance in the overall accuracy.
