  Pretrained protein structure models without labels are crucial foundations
for the majority of protein downstream applications. The conventional structure
pretraining methods follow the mature natural language pretraining methods such
as denoised reconstruction and masked language modeling but usually destroy the
real representation of spatial structures. The other common pretraining methods
might predict a fixed set of predetermined object categories, where a
restricted supervised manner limits their generality and usability as
additional labeled data is required to specify any other protein concepts. In
this work, we introduce a novel unsupervised protein structure representation
pretraining with a robust protein language model. In particular, we first
propose to leverage an existing pretrained language model to guide structure
model learning through an unsupervised contrastive alignment. In addition, a
self-supervised structure constraint is proposed to further learn the intrinsic
information about the structures. With only light training data, the pretrained
structure model can obtain better generalization ability. To quantitatively
evaluate the proposed structure models, we design a series of rational
evaluation methods, including internal tasks (e.g., contact map prediction,
distribution alignment quality) and external/downstream tasks (e.g., protein
design). The extensive experimental results conducted on multiple tasks and
specific datasets demonstrate the superiority of the proposed
sequence-structure transformation framework.
