  Natural Language Processing (NLP) has recently gained wide attention in
cybersecurity, particularly in Cyber Threat Intelligence (CTI) and cyber
automation. Increased connection and automation have revolutionized the world's
economic and cultural infrastructures, while they have introduced risks in
terms of cyber attacks. CTI is information that helps cybersecurity analysts
make intelligent security decisions, that is often delivered in the form of
natural language text, which must be transformed to machine readable format
through an automated procedure before it can be used for automated security
measures.
  This paper proposes SecureBERT, a cybersecurity language model capable of
capturing text connotations in cybersecurity text (e.g., CTI) and therefore
successful in automation for many critical cybersecurity tasks that would
otherwise rely on human expertise and time-consuming manual efforts. SecureBERT
has been trained using a large corpus of cybersecurity text.To make SecureBERT
effective not just in retaining general English understanding, but also when
applied to text with cybersecurity implications, we developed a customized
tokenizer as well as a method to alter pre-trained weights. The SecureBERT is
evaluated using the standard Masked Language Model (MLM) test as well as two
additional standard NLP tasks. Our evaluation studies show that
SecureBERT\footnote{\url{https://github.com/ehsanaghaei/SecureBERT}}
outperforms existing similar models, confirming its capability for solving
crucial NLP tasks in cybersecurity.
