  Recent advancement in large language models (LLMs) has offered a strong
potential for natural language systems to process informal language. A
representative form of informal language is slang, used commonly in daily
conversations and online social media. To date, slang has not been
comprehensively evaluated in LLMs due partly to the absence of a carefully
designed and publicly accessible benchmark. Using movie subtitles, we construct
a dataset that supports evaluation on a diverse set of tasks pertaining to
automatic processing of slang. For both evaluation and finetuning, we show the
effectiveness of our dataset on two core applications: 1) slang detection, and
2) identification of regional and historical sources of slang from natural
sentences. We also show how our dataset can be used to probe the output
distributions of LLMs for interpretive insights. We find that while LLMs such
as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like
models finetuned on our dataset achieve comparable performance. Furthermore, we
show that our dataset enables finetuning of LLMs such as GPT-3.5 that achieve
substantially better performance than strong zero-shot baselines. Our work
offers a comprehensive evaluation and a high-quality benchmark on English slang
based on the OpenSubtitles corpus, serving both as a publicly accessible
resource and a platform for applying tools for informal language processing.
