  High-quality and large-scale data are key to success for AI systems. However,
large-scale data annotation efforts are often confronted with a set of common
challenges: (1) designing a user-friendly annotation interface; (2) training
enough annotators efficiently; and (3) reproducibility. To address these
problems, we introduce Crowdaq, an open-source platform that standardizes the
data collection pipeline with customizable user-interface components, automated
annotator qualification, and saved pipelines in a re-usable format. We show
that Crowdaq simplifies data annotation significantly on a diverse set of data
collection use cases and we hope it will be a convenient tool for the
community.
