  Twitter is a well-known microblogging social site where users express their
views and opinions in real-time. As a result, tweets tend to contain valuable
information. With the advancements of deep learning in the domain of natural
language processing, extracting meaningful information from tweets has become a
growing interest among natural language researchers. Applying existing language
representation models to extract information from Twitter does not often
produce good results. Moreover, there is no existing language representation
models for text analysis specific to the social media domain. Hence, in this
article, we introduce two TweetBERT models, which are domain specific language
presentation models, pre-trained on millions of tweets. We show that the
TweetBERT models significantly outperform the traditional BERT models in
Twitter text mining tasks by more than 7% on each Twitter dataset. We also
provide an extensive analysis by evaluating seven BERT models on 31 different
datasets. Our results validate our hypothesis that continuously training
language models on twitter corpus help performance with Twitter.
