  Hypergraphs are characterized by complex topological structure, representing
higher-order interactions among multiple entities through hyperedges. Lately,
hypergraph-based deep learning methods to learn informative data
representations for the problem of node classification on text-attributed
hypergraphs have garnered increasing research attention. However, existing
methods struggle to simultaneously capture the full extent of hypergraph
structural information and the rich linguistic attributes inherent in the nodes
attributes, which largely hampers their effectiveness and generalizability. To
overcome these challenges, we explore ways to further augment a pretrained BERT
model with specialized hypergraph-aware layers for the task of node
classification. Such layers introduce higher-order structural inductive bias
into the language model, thus improving the model's capacity to harness both
higher-order context information from the hypergraph structure and semantic
information present in text. In this paper, we propose a new architecture,
HyperBERT, a mixed text-hypergraph model which simultaneously models hypergraph
relational structure while maintaining the high-quality text encoding
capabilities of a pre-trained BERT. Notably, HyperBERT presents results that
achieve a new state-of-the-art on five challenging text-attributed hypergraph
node classification benchmarks.
