  End-to-end (E2E) spoken language understanding (SLU) systems predict
utterance semantics directly from speech using a single model. Previous work in
this area has focused on targeted tasks in fixed domains, where the output
semantic structure is assumed a priori and the input speech is of limited
complexity. In this work we present our approach to developing an E2E model for
generalized SLU in commercial voice assistants (VAs). We propose a fully
differentiable, transformer-based, hierarchical system that can be pretrained
at both the ASR and NLU levels. This is then fine-tuned on both transcription
and semantic classification losses to handle a diverse set of intent and
argument combinations. This leads to an SLU system that achieves significant
improvements over baselines on a complex internal generalized VA dataset with a
43% improvement in accuracy, while still meeting the 99% accuracy benchmark on
the popular Fluent Speech Commands dataset. We further evaluate our model on a
hard test set, exclusively containing slot arguments unseen in training, and
demonstrate a nearly 20% improvement, showing the efficacy of our approach in
truly demanding VA scenarios.
