  This paper introduces a human-in-the-loop (HITL) data annotation pipeline to
generate high-quality, large-scale speech datasets. The pipeline combines human
and machine advantages to more quickly, accurately, and cost-effectively
annotate datasets with machine pre-labeling and fully manual auditing. Quality
control mechanisms such as blind testing, behavior monitoring, and data
validation have been adopted in the annotation pipeline to mitigate potential
bias introduced by machine-generated labels. Our A/B testing and pilot results
demonstrated the HITL pipeline can improve annotation speed and capacity by at
least 80% and quality is comparable to or higher than manual double pass
annotation. We are leveraging this scalable pipeline to create and continuously
grow ultra-high volume off-the-shelf (UHV-OTS) speech corpora for multiple
languages, with the capability to expand to 10,000+ hours per language
annually. Customized datasets can be produced from the UHV-OTS corpora using
dynamic packaging. UHV-OTS is a long-term Appen project to support commercial
and academic research data needs in speech processing. Appen will donate a
number of free speech datasets from the UHV-OTS each year to support academic
and open source community research under the CC-BY-SA license. We are also
releasing the code of the data pre-processing and pre-tagging pipeline under
the Apache 2.0 license to allow reproduction of the results reported in the
paper.
