  Code-switching is the phenomenon by which bilingual speakers switch between
multiple languages during communication. The importance of developing language
technologies for codeswitching data is immense, given the large populations
that routinely code-switch. High-quality linguistic annotations are extremely
valuable for any NLP task, and performance is often limited by the amount of
high-quality labeled data. However, little such data exists for code-switching.
In this paper, we describe crowd-sourcing universal part-of-speech tags for the
Miami Bangor Corpus of Spanish-English code-switched speech. We split the
annotation task into three subtasks: one in which a subset of tokens are
labeled automatically, one in which questions are specifically designed to
disambiguate a subset of high frequency words, and a more general cascaded
approach for the remaining data in which questions are displayed to the worker
following a decision tree structure. Each subtask is extended and adapted for a
multilingual setting and the universal tagset. The quality of the annotation
process is measured using hidden check questions annotated with gold labels.
The overall agreement between gold standard labels and the majority vote is
between 0.95 and 0.96 for just three labels and the average recall across
part-of-speech tags is between 0.87 and 0.99, depending on the task.
