  Researchers have traditionally recruited native speakers to provide
annotations for widely used benchmark datasets. However, there are languages
for which recruiting native speakers can be difficult, and it would help to
find learners of those languages to annotate the data. In this paper, we
investigate whether language learners can contribute annotations to benchmark
datasets. In a carefully controlled annotation experiment, we recruit 36
language learners, provide two types of additional resources (dictionaries and
machine-translated sentences), and perform mini-tests to measure their language
proficiency. We target three languages, English, Korean, and Indonesian, and
the four NLP tasks of sentiment analysis, natural language inference, named
entity recognition, and machine reading comprehension. We find that language
learners, especially those with intermediate or advanced levels of language
proficiency, are able to provide fairly accurate labels with the help of
additional resources. Moreover, we show that data annotation improves learners'
language proficiency in terms of vocabulary and grammar. One implication of our
findings is that broadening the annotation task to include language learners
can open up the opportunity to build benchmark datasets for languages for which
it is difficult to recruit native speakers.
