  While word error rates of automatic speech recognition (ASR) systems have
consistently fallen, natural language understanding (NLU) applications built on
top of ASR systems still attribute significant numbers of failures to
low-quality speech recognition results. Existing assistant systems collect
large numbers of these unsuccessful interactions, but these systems usually
fail to learn from these interactions, even in an offline fashion. In this
work, we introduce CLC: Contrastive Learning for Conversations, a family of
methods for contrastive fine-tuning of models in a self-supervised fashion,
making use of easily detectable artifacts in unsuccessful conversations with
assistants. We demonstrate that our CLC family of approaches can improve the
performance of ASR models on OD3, a new public large-scale semi-synthetic
meta-dataset of audio task-oriented dialogues, by up to 19.2%. These gains
transfer to real-world systems as well, where we show that CLC can help to
improve performance by up to 6.7% over baselines. We make OD3 publicly
available at https://github.com/amazon-science/amazon-od3 .
