  We study the ability of transformer-based language models (LMs) to understand
social media language. Social media (SM) language is distinct from standard
written language, yet existing benchmarks fall short of capturing LM
performance in this socially, economically, and politically important domain.
We quantify the degree to which social media language differs from conventional
language and conclude that the difference is significant both in terms of token
distribution and rate of linguistic shift. Next, we introduce a new benchmark
for Social MedIa Language Evaluation (SMILE) that covers four SM platforms and
eleven tasks. Finally, we show that learning a tokenizer and pretraining on a
mix of social media and conventional language yields an LM that outperforms the
best similar-sized alternative by 4.2 points on the overall SMILE score.
