  Self-supervised learning (SSL) for speech representation has been
successfully applied in various downstream tasks, such as speech and speaker
recognition. More recently, speech SSL models have also been shown to be
beneficial in advancing spoken language understanding tasks, implying that the
SSL models have the potential to learn not only acoustic but also linguistic
information. In this paper, we aim to clarify if speech SSL techniques can well
capture linguistic knowledge. For this purpose, we introduce SpeechGLUE, a
speech version of the General Language Understanding Evaluation (GLUE)
benchmark. Since GLUE comprises a variety of natural language understanding
tasks, SpeechGLUE can elucidate the degree of linguistic ability of speech SSL
models. Experiments demonstrate that speech SSL models, although inferior to
text-based SSL models, perform better than baselines, suggesting that they can
acquire a certain amount of general linguistic knowledge from just unlabeled
speech data.
