  Graph neural network models have been extensively used to learn node
representations for graph structured data in an end-to-end setting. These
models often rely on localized first order approximations of spectral graph
convolutions and hence are unable to capture higher-order relational
information between nodes. Probabilistic Graphical Models form another class of
models that provide rich flexibility in incorporating such relational
information but are limited by inefficient approximate inference algorithms at
higher order. In this paper, we propose to combine these approaches to learn
better node and graph representations. First, we derive an efficient
approximate sum-product loopy belief propagation inference algorithm for
higher-order PGMs. We then embed the message passing updates into a neural
network to provide the inductive bias of the inference algorithm in end-to-end
learning. This gives us a model that is flexible enough to accommodate domain
knowledge while maintaining the computational advantage. We further propose
methods for constructing higher-order factors that are conditioned on node and
edge features and share parameters wherever necessary. Our experimental
evaluation shows that our model indeed captures higher-order information,
substantially outperforming state-of-the-art $k$-order graph neural networks in
molecular datasets.
