  We present GEST -- a new dataset for measuring gender-stereotypical reasoning
in masked LMs and English-to-X machine translation systems. GEST contains
samples that are compatible with 9 Slavic languages and English for 16 gender
stereotypes about men and women (e.g., Women are beautiful, Men are leaders).
The definition of said stereotypes was informed by gender experts. We used GEST
to evaluate 11 masked LMs and 4 machine translation systems. We discovered
significant and consistent amounts of stereotypical reasoning in almost all the
evaluated models and languages.
