  The irreducible complexity of natural phenomena has led Graph Neural Networks
to be employed as a standard model to perform representation learning tasks on
graph-structured data. While their capacity to capture local and global
patterns is remarkable, the implications associated with long-range and
higher-order dependencies pose considerable challenges to such models. This
work starts with a theoretical framework to reveal the impact of network's
width, depth, and graph topology on the over-squashing phenomena in
message-passing neural networks. Then, the work drifts towards, higher-order
interactions and multi-relational inductive biases via Topological Neural
Networks. Such models propagate messages through higher-dimensional structures,
providing shortcuts or additional routes for information flow. With this
construction, the underlying computational graph is no longer coupled with the
input graph structure, thus mitigating the aforementioned bottlenecks while
accounting also for higher-order interactions. Inspired by Graph Attention
Networks, two topological attention networks are proposed: Simplicial and Cell
Attention Networks. The rationale behind these architecture is to leverage the
extended notion of neighbourhoods provided by the arrangement of groups of
nodes within a simplicial or cell complex to design anisotropic aggregations
able to measure the importance of the information coming from different regions
of the domain. By doing so, they capture dependencies that conventional Graph
Neural Networks might miss. Finally, a multi-way communication scheme is
introduced with Enhanced Cellular Isomorphism Networks, which augment
topological message passing schemes to enable a direct interactions among
groups of nodes arranged in ring-like structures.
