  Human language can be expressed in either written or spoken form, i.e. text
or speech. Humans can acquire knowledge from text to improve speaking and
listening. However, the quest for speech pre-trained models to leverage
unpaired text has just started. In this paper, we investigate a new way to
pre-train such a joint speech-text model to learn enhanced speech
representations and benefit various speech-related downstream tasks.
Specifically, we propose a novel pre-training method, text-guided HuBERT, or
T-HuBERT, which performs self-supervised learning over speech to derive
phoneme-like discrete representations. And these phoneme-like pseudo-label
sequences are firstly derived from speech via the generative adversarial
networks (GAN) to be statistically similar to those from additional unpaired
textual data. In this way, we build a bridge between unpaired speech and text
in an unsupervised manner. Extensive experiments demonstrate the significant
superiority of our proposed method over various strong baselines, which
achieves up to 15.3% relative Word Error Rate (WER) reduction on the
LibriSpeech dataset.
