  Representation learning for proteins has primarily focused on the global
understanding of protein sequences regardless of their length. However, shorter
proteins (known as peptides) take on distinct structures and functions compared
to their longer counterparts. Unfortunately, there are not as many naturally
occurring peptides available to be sequenced and therefore less
peptide-specific data to train with. In this paper, we propose a new peptide
data augmentation scheme, where we train peptide language models on
artificially constructed peptides that are small contiguous subsets of longer,
wild-type proteins; we refer to the training peptides as "chopped proteins". We
evaluate the representation potential of models trained with chopped proteins
versus natural peptides and find that training language models with chopped
proteins results in more generalized embeddings for short protein sequences.
These peptide-specific models also retain information about the original
protein they were derived from better than language models trained on
full-length proteins. We compare masked language model training objectives to
three novel peptide-specific training objectives: next-peptide prediction,
contrastive peptide selection and evolution-weighted MLM. We demonstrate
improved zero-shot learning performance for a deep mutational scan peptides
benchmark.
