  Self-supervised speech representations have been shown to be effective in a
variety of speech applications. However, existing representation learning
methods generally rely on the autoregressive model and/or observed global
dependencies while generating the representation. In this work, we propose
Non-Autoregressive Predictive Coding (NPC), a self-supervised method, to learn
a speech representation in a non-autoregressive manner by relying only on local
dependencies of speech. NPC has a conceptually simple objective and can be
implemented easily with the introduced Masked Convolution Blocks. NPC offers a
significant speedup for inference since it is parallelizable in time and has a
fixed inference time for each time step regardless of the input sequence
length. We discuss and verify the effectiveness of NPC by theoretically and
empirically comparing it with other methods. We show that the NPC
representation is comparable to other methods in speech experiments on phonetic
and speaker classification while being more efficient.
