  Gender bias is largely recognized as a problematic phenomenon affecting
language technologies, with recent studies underscoring that it might surface
differently across languages. However, most of current evaluation practices
adopt a word-level focus on a narrow set of occupational nouns under synthetic
conditions. Such protocols overlook key features of grammatical gender
languages, which are characterized by morphosyntactic chains of gender
agreement, marked on a variety of lexical items and parts-of-speech (POS). To
overcome this limitation, we enrich the natural, gender-sensitive MuST-SHE
corpus (Bentivogli et al., 2020) with two new linguistic annotation layers (POS
and agreement chains), and explore to what extent different lexical categories
and agreement phenomena are impacted by gender skews. Focusing on speech
translation, we conduct a multifaceted evaluation on three language directions
(English-French/Italian/Spanish), with models trained on varying amounts of
data and different word segmentation techniques. By shedding light on model
behaviours, gender bias, and its detection at several levels of granularity,
our findings emphasize the value of dedicated analyses beyond aggregated
overall results.
