  Neural Machine Translation (NMT) models, though state-of-the-art for
translation, often reflect social biases, particularly gender bias. Existing
evaluation benchmarks primarily focus on English as the source language of
translation. For source languages other than English, studies often employ
gender-neutral sentences for bias evaluation, whereas real-world sentences
frequently contain gender information in different forms. Therefore, it makes
more sense to evaluate for bias using such source sentences to determine if NMT
models can discern gender from the grammatical gender cues rather than relying
on biased associations. To illustrate this, we create two gender-specific
sentence sets in Hindi to automatically evaluate gender bias in various
Hindi-English (HI-EN) NMT systems. We emphasise the significance of tailoring
bias evaluation test sets to account for grammatical gender markers in the
source language.
