  In recent years, a series of Transformer-based models unlocked major
improvements in general natural language understanding (NLU) tasks. Such a fast
pace of research would not be possible without general NLU benchmarks, which
allow for a fair comparison of the proposed methods. However, such benchmarks
are available only for a handful of languages. To alleviate this issue, we
introduce a comprehensive multi-task benchmark for the Polish language
understanding, accompanied by an online leaderboard. It consists of a diverse
set of tasks, adopted from existing datasets for named entity recognition,
question-answering, textual entailment, and others. We also introduce a new
sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR).
To ensure a common evaluation scheme and promote models that generalize to
different NLU tasks, the benchmark includes datasets from varying domains and
applications. Additionally, we release HerBERT, a Transformer-based model
trained specifically for the Polish language, which has the best average
performance and obtains the best results for three out of nine tasks. Finally,
we provide an extensive evaluation, including several standard baselines and
recently proposed, multilingual Transformer-based models.
