  Recently, numerous embedding models have been made available and widely used
for various NLP tasks. The Massive Text Embedding Benchmark (MTEB) has
primarily simplified the process of choosing a model that performs well for
several tasks in English, but extensions to other languages remain challenging.
This is why we expand MTEB to propose the first massive benchmark of sentence
embeddings for French. We gather 15 existing datasets in an easy-to-use
interface and create three new French datasets for a global evaluation of 8
task categories. We compare 51 carefully selected embedding models on a large
scale, conduct comprehensive statistical tests, and analyze the correlation
between model performance and many of their characteristics. We find out that
even if no model is the best on all tasks, large multilingual models
pre-trained on sentence similarity perform exceptionally well. Our work comes
with open-source code, new datasets and a public leaderboard.
