  The successful application of neural methods to machine translation has
realized huge quality advances for the community. With these improvements, many
have noted outstanding challenges, including the modeling and treatment of
gendered language. While previous studies have identified issues using
synthetic examples, we develop a novel technique to mine examples from real
world data to explore challenges for deployed systems. We use our method to
compile an evaluation benchmark spanning examples for four languages from three
language families, which we publicly release to facilitate research. The
examples in our benchmark expose where model representations are gendered, and
the unintended consequences these gendered representations can have in
downstream application.
