  Cybersecurity information is often technically complex and relayed through
unstructured text, making automation of cyber threat intelligence highly
challenging. For such text domains that involve high levels of expertise,
pretraining on in-domain corpora has been a popular method for language models
to obtain domain expertise. However, cybersecurity texts often contain
non-linguistic elements (such as URLs and hash values) that could be unsuitable
with the established pretraining methodologies. Previous work in other domains
have removed or filtered such text as noise, but the effectiveness of these
methods have not been investigated, especially in the cybersecurity domain. We
propose different pretraining methodologies and evaluate their effectiveness
through downstream tasks and probing tasks. Our proposed strategy (selective
MLM and jointly training NLE token classification) outperforms the commonly
taken approach of replacing non-linguistic elements (NLEs). We use our
domain-customized methodology to train CyBERTuned, a cybersecurity domain
language model that outperforms other cybersecurity PLMs on most tasks.
