  Learning meaningful and general representations from unannotated speech that
are applicable to a wide range of tasks remains challenging. In this paper we
propose to use autoregressive predictive coding (APC), a recently proposed
self-supervised objective, as a generative pre-training approach for learning
meaningful, non-specific, and transferable speech representations. We pre-train
APC on large-scale unlabeled data and conduct transfer learning experiments on
three speech applications that require different information about speech
characteristics to perform well: speech recognition, speech translation, and
speaker identification. Extensive experiments show that APC not only
outperforms surface features (e.g., log Mel spectrograms) and other popular
representation learning methods on all three tasks, but is also effective at
reducing downstream labeled data size and model parameters. We also investigate
the use of Transformers for modeling APC and find it superior to RNNs.
