  While multilingual language models can improve NLP performance on
low-resource languages by leveraging higher-resource languages, they also
reduce average performance on all languages (the 'curse of multilinguality').
Here we show another problem with multilingual models: grammatical structures
in higher-resource languages bleed into lower-resource languages, a phenomenon
we call grammatical structure bias. We show this bias via a novel method for
comparing the fluency of multilingual models to the fluency of monolingual
Spanish and Greek models: testing their preference for two carefully-chosen
variable grammatical structures (optional pronoun-drop in Spanish and optional
Subject-Verb ordering in Greek). We find that multilingual BERT is biased
toward the English-like setting (explicit pronouns and Subject-Verb-Object
ordering) as compared to our monolingual control language model. With our case
studies, we hope to bring to light the fine-grained ways in which multilingual
models can be biased,and encourage more linguistically-aware fluency
evaluation.
