  We present Polish Information Retrieval Benchmark (PIRB), a comprehensive
evaluation framework encompassing 41 text information retrieval tasks for
Polish. The benchmark incorporates existing datasets as well as 10 new,
previously unpublished datasets covering diverse topics such as medicine, law,
business, physics, and linguistics. We conduct an extensive evaluation of over
20 dense and sparse retrieval models, including the baseline models trained by
us as well as other available Polish and multilingual methods. Finally, we
introduce a three-step process for training highly effective language-specific
retrievers, consisting of knowledge distillation, supervised fine-tuning, and
building sparse-dense hybrid retrievers using a lightweight rescoring model. In
order to validate our approach, we train new text encoders for Polish and
compare their results with previously evaluated methods. Our dense models
outperform the best solutions available to date, and the use of hybrid methods
further improves their performance.
