  Much recent work on Spoken Language Understanding (SLU) falls short in at
least one of three ways: models were trained on oracle text input and neglected
the Automatics Speech Recognition (ASR) outputs, models were trained to predict
only intents without the slot values, or models were trained on a large amount
of in-house data. We proposed a clean and general framework to learn semantics
directly from speech with semi-supervision from transcribed speech to address
these. Our framework is built upon pretrained end-to-end (E2E) ASR and
self-supervised language models, such as BERT, and fine-tuned on a limited
amount of target SLU corpus. In parallel, we identified two inadequate settings
under which SLU models have been tested: noise-robustness and E2E semantics
evaluation. We tested the proposed framework under realistic environmental
noises and with a new metric, the slots edit F1 score, on two public SLU
corpora. Experiments show that our SLU framework with speech as input can
perform on par with those with oracle text as input in semantics understanding,
while environmental noises are present, and a limited amount of labeled
semantics data is available.
