  Despite its relevance, the maturity of NLP for social media pales in
comparison with general-purpose models, metrics and benchmarks. This fragmented
landscape makes it hard for the community to know, for instance, given a task,
which is the best performing model and how it compares with others. To
alleviate this issue, we introduce a unified benchmark for NLP evaluation in
social media, SuperTweetEval, which includes a heterogeneous set of tasks and
datasets combined, adapted and constructed from scratch. We benchmarked the
performance of a wide range of models on SuperTweetEval and our results suggest
that, despite the recent advances in language modelling, social media remains
challenging.
