  Much recent work on Spoken Language Understanding (SLU) is limited in at
least one of three ways: models were trained on oracle text input and neglected
ASR errors, models were trained to predict only intents without the slot
values, or models were trained on a large amount of in-house data. In this
paper, we propose a clean and general framework to learn semantics directly
from speech with semi-supervision from transcribed or untranscribed speech to
address these issues. Our framework is built upon pretrained end-to-end (E2E)
ASR and self-supervised language models, such as BERT, and fine-tuned on a
limited amount of target SLU data. We study two semi-supervised settings for
the ASR component: supervised pretraining on transcribed speech, and
unsupervised pretraining by replacing the ASR encoder with self-supervised
speech representations, such as wav2vec. In parallel, we identify two essential
criteria for evaluating SLU models: environmental noise-robustness and E2E
semantics evaluation. Experiments on ATIS show that our SLU framework with
speech as input can perform on par with those using oracle text as input in
semantics understanding, even though environmental noise is present and a
limited amount of labeled semantics data is available for training.
