  Language identification is an important first step in many IR and NLP
applications. Most publicly available language identification datasets,
however, are compiled under the assumption that the gold label of each instance
is determined by where texts are retrieved from. Research has shown that this
is a problematic assumption, particularly in the case of very similar languages
(e.g., Croatian and Serbian) and national language varieties (e.g., Brazilian
and European Portuguese), where texts may contain no distinctive marker of the
particular language or variety. To overcome this important limitation, this
paper presents DSL True Labels (DSL-TL), the first human-annotated multilingual
dataset for language variety identification. DSL-TL contains a total of 12,900
instances in Portuguese, split between European Portuguese and Brazilian
Portuguese; Spanish, split between Argentine Spanish and Castilian Spanish; and
English, split between American English and British English. We trained
multiple models to discriminate between these language varieties, and we
present the results in detail. The data and models presented in this paper
provide a reliable benchmark toward the development of robust and fairer
language variety identification systems. We make DSL-TL freely available to the
research community.
