  Recent work has shown that Pre-trained Language Models (PLMs) store the
relational knowledge learned from data and utilize it for performing downstream
tasks. However, commonsense knowledge across different regions may vary. For
instance, the color of bridal dress is white in American weddings whereas it is
red in Chinese weddings. In this paper, we introduce a benchmark dataset,
Geo-Diverse Commonsense Multilingual Language Models Analysis (GeoMLAMA), for
probing the diversity of the relational knowledge in multilingual PLMs.
GeoMLAMA contains 3,125 prompts in English, Chinese, Hindi, Persian, and
Swahili, with a wide coverage of concepts shared by people from American,
Chinese, Indian, Iranian and Kenyan cultures. We benchmark 11 standard
multilingual PLMs on GeoMLAMA. Interestingly, we find that 1) larger
multilingual PLMs variants do not necessarily store geo-diverse concepts better
than its smaller variant; 2) multilingual PLMs are not intrinsically biased
towards knowledge from the Western countries (the United States); 3) the native
language of a country may not be the best language to probe its knowledge and
4) a language may better probe knowledge about a non-native country than its
native country. Code and data are released at
https://github.com/WadeYin9712/GeoMLAMA.
