  Beyond individual languages, multilingual natural language processing (NLP)
research increasingly aims to develop models that perform well across languages
generally. However, evaluating these systems on all the world's languages is
practically infeasible. To attain generalizability, representative language
sampling is essential. Previous work argues that generalizable multilingual
evaluation sets should contain languages with diverse typological properties.
However, 'typologically diverse' language samples have been found to vary
considerably in this regard, and popular sampling methods are flawed and
inconsistent. We present a language sampling framework for selecting highly
typologically diverse languages given a sampling frame, informed by language
typology. We compare sampling methods with a range of metrics and find that our
systematic methods consistently retrieve more typologically diverse language
selections than previous methods in NLP. Moreover, we provide evidence that
this affects generalizability in multilingual model evaluation, emphasizing the
importance of diverse language sampling in NLP evaluation.
