  Gender bias has been a focal point in the study of bias in machine
translation and language models. Existing machine translation gender bias
evaluations are primarily focused on male and female genders, limiting the
scope of the evaluation. To assess gender bias accurately, these studies often
rely on calculating the accuracy of gender pronouns or the masculine and
feminine attributes of grammatical gender via the stereotypes triggered by
occupations or sentiment words ({\em i.e.}, clear positive or negative
attitude), which cannot extend to non-binary groups. This study presents a
benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude
words), which assesses gender bias beyond binary gender. Meanwhile, we propose
a novel process to evaluate gender bias based on the Emotional Attitude Score
(EAS), which is used to quantify ambiguous attitude words. In evaluating three
recent and effective open-source LLMs and one powerful multilingual
translation-specific model, our main observations are: (1) The translation
performance within non-binary gender contexts is markedly inferior in terms of
translation quality and exhibits more negative attitudes than binary-gender
contexts. (2) The analysis experiments indicate that incorporating constraint
context in prompts for gender identity terms can substantially reduce
translation bias, while the bias remains evident despite the presence of the
constraints. The code is publicly available at
\url{https://github.com/pppa2019/ambGIMT}.
