  Recent advances in Language Models have enabled the protein modeling
community with a powerful tool since protein sequences can be represented as
text. Specifically, by taking advantage of Transformers, sequence-to-property
prediction will be amenable without the need for explicit structural data. In
this work, inspired by recent progress in Large Language Models (LLMs), we
introduce PeptideBERT, a protein language model for predicting three key
properties of peptides (hemolysis, solubility, and non-fouling). The
PeptideBert utilizes the ProtBERT pretrained transformer model with 12
attention heads and 12 hidden layers. We then finetuned the pretrained model
for the three downstream tasks. Our model has achieved state of the art (SOTA)
for predicting Hemolysis, which is a task for determining peptide's potential
to induce red blood cell lysis. Our PeptideBert non-fouling model also achieved
remarkable accuracy in predicting peptide's capacity to resist non-specific
interactions. This model, trained predominantly on shorter sequences, benefits
from the dataset where negative examples are largely associated with insoluble
peptides. Codes, models, and data used in this study are freely available at:
https://github.com/ChakradharG/PeptideBERT
