  In this work, we assess the ability of foundation models to recall
encyclopedic knowledge across a wide range of linguistic contexts. To support
this, we: 1) produce a 20-language dataset that contains 303k factual
associations paired with counterfactuals, 2) evaluate 5 models in a
multilingual test, and 3) benchmark a diverse set of 24 models in an
English-only test. Meta's LLaMA achieves the highest scores in both
multilingual and English-only evaluations. Yet, an analysis of LLaMA's errors
reveals significant limitations in its ability to recall facts in languages
other than English, plus difficulties related to the location and gender of
fact subjects. Overall, our findings suggest that today's foundation models are
far from polyglots.
