  End-to-end Spoken Language Understanding (E2E SLU) has attracted increasing
interest due to its advantages of joint optimization and low latency when
compared to traditionally cascaded pipelines. Existing E2E SLU models usually
follow a two-stage configuration where an Automatic Speech Recognition (ASR)
network first predicts a transcript which is then passed to a Natural Language
Understanding (NLU) module through an interface to infer semantic labels, such
as intent and slot tags. This design, however, does not consider the NLU
posterior while making transcript predictions, nor correct the NLU prediction
error immediately by considering the previously predicted word-pieces. In
addition, the NLU model in the two-stage system is not streamable, as it must
wait for the audio segments to complete processing, which ultimately impacts
the latency of the SLU system. In this work, we propose a streamable multi-task
semantic transducer model to address these considerations. Our proposed
architecture predicts ASR and NLU labels auto-regressively and uses a semantic
decoder to ingest both previously predicted word-pieces and slot tags while
aggregating them through a fusion network. Using an industry scale SLU and a
public FSC dataset, we show the proposed model outperforms the two-stage E2E
SLU model for both ASR and NLU metrics.
