  Protein representation learning aims to learn informative protein embeddings
capable of addressing crucial biological questions, such as protein function
prediction. Although sequence-based transformer models have shown promising
results by leveraging the vast amount of protein sequence data in a
self-supervised way, there is still a gap in applying these methods to 3D
protein structures. In this work, we propose a pre-training scheme going beyond
trivial masking methods leveraging 3D and hierarchical structures of proteins.
We propose a novel self-supervised method to pretrain 3D graph neural networks
on 3D protein structures, by predicting the distances between local geometric
centroids of protein subgraphs and the global geometric centroid of the
protein. The motivation for this method is twofold. First, the relative spatial
arrangements and geometric relationships among different regions of a protein
are crucial for its function. Moreover, proteins are often organized in a
hierarchical manner, where smaller substructures, such as secondary structure
elements, assemble into larger domains. By considering subgraphs and their
relationships to the global protein structure, the model can learn to reason
about these hierarchical levels of organization. We experimentally show that
our proposed pertaining strategy leads to significant improvements in the
performance of 3D GNNs in various protein classification tasks.
