  The latest advancements in machine learning and deep learning have brought
forth the concept of semantic similarity, which has proven immensely beneficial
in multiple applications and has largely replaced keyword search. However,
evaluating semantic similarity and conducting searches for a specific query
across various documents continue to be a complicated task. This complexity is
due to the multifaceted nature of the task, the lack of standard benchmarks,
whereas these challenges are further amplified for Arabic language. This paper
endeavors to establish a straightforward yet potent benchmark for semantic
search in Arabic. Moreover, to precisely evaluate the effectiveness of these
metrics and the dataset, we conduct our assessment of semantic search within
the framework of retrieval augmented generation (RAG).
