  Neural Machine Translation (NMT) models are state-of-the-art for machine
translation. However, these models are known to have various social biases,
especially gender bias. Most of the work on evaluating gender bias in NMT has
focused primarily on English as the source language. For source languages
different from English, most of the studies use gender-neutral sentences to
evaluate gender bias. However, practically, many sentences that we encounter do
have gender information. Therefore, it makes more sense to evaluate for bias
using such sentences. This allows us to determine if NMT models can identify
the correct gender based on the grammatical gender cues in the source sentence
rather than relying on biased correlations with, say, occupation terms. To
demonstrate our point, in this work, we use Hindi as the source language and
construct two sets of gender-specific sentences: OTSC-Hindi and WinoMT-Hindi
that we use to evaluate different Hindi-English (HI-EN) NMT systems
automatically for gender bias. Our work highlights the importance of
considering the nature of language when designing such extrinsic bias
evaluation datasets.
