  We propose vq-wav2vec to learn discrete representations of audio segments
through a wav2vec-style self-supervised context prediction task. The algorithm
uses either a gumbel softmax or online k-means clustering to quantize the dense
representations. Discretization enables the direct application of algorithms
from the NLP community which require discrete inputs. Experiments show that
BERT pre-training achieves a new state of the art on TIMIT phoneme
classification and WSJ speech recognition.
