  Spoken language understanding (SLU) is a task aiming to extract high-level
semantics from spoken utterances. Previous works have investigated the use of
speech self-supervised models and textual pre-trained models, which have shown
reasonable improvements to various SLU tasks. However, because of the
mismatched modalities between speech signals and text tokens, previous methods
usually need complex designs of the frameworks. This work proposes a simple yet
efficient unsupervised paradigm that connects speech and textual pre-trained
models, resulting in an unsupervised speech-to-semantic pre-trained model for
various tasks in SLU. To be specific, we propose to use unsupervised automatic
speech recognition (ASR) as a connector that bridges different modalities used
in speech and textual pre-trained models. Our experiments show that
unsupervised ASR itself can improve the representations from speech
self-supervised models. More importantly, it is shown as an efficient connector
between speech and textual pre-trained models, improving the performances of
five different SLU tasks. Notably, on spoken question answering, we reach the
state-of-the-art result over the challenging NMSQA benchmark.
