  While large language models (LLMs) excel in various natural language tasks in
English, their performance in lower-resourced languages like Hebrew, especially
for generative tasks such as abstractive summarization, remains unclear. The
high morphological richness in Hebrew adds further challenges due to the
ambiguity in sentence comprehension and the complexities in meaning
construction. In this paper, we address this resource and evaluation gap by
introducing HeSum, a novel benchmark specifically designed for abstractive text
summarization in Modern Hebrew. HeSum consists of 10,000 article-summary pairs
sourced from Hebrew news websites written by professionals. Linguistic analysis
confirms HeSum's high abstractness and unique morphological challenges. We show
that HeSum presents distinct difficulties for contemporary state-of-the-art
LLMs, establishing it as a valuable testbed for generative language technology
in Hebrew, and MRLs generative challenges in general.
