  In this paper, we introduce the Polish Massive Text Embedding Benchmark
(PL-MTEB), a comprehensive benchmark for text embeddings in Polish. The PL-MTEB
consists of 28 diverse NLP tasks from 5 task types. We adapted the tasks based
on previously used datasets by the Polish NLP community. In addition, we
created a new PLSC (Polish Library of Science Corpus) dataset consisting of
titles and abstracts of scientific publications in Polish, which was used as
the basis for two novel clustering tasks. We evaluated 15 publicly available
models for text embedding, including Polish and multilingual ones, and
collected detailed results for individual tasks and aggregated results for each
task type and the entire benchmark. PL-MTEB comes with open-source code at
https://github.com/rafalposwiata/pl-mteb.
