  Accurate prediction of the user intent to interact with a voice assistant
(VA) on a device (e.g. on the phone) is critical for achieving naturalistic,
engaging, and privacy-centric interactions with the VA. To this end, we present
a novel approach to predict the user's intent (the user speaking to the device
or not) directly from acoustic and textual information encoded at subword
tokens which are obtained via an end-to-end ASR model. Modeling directly the
subword tokens, compared to modeling of the phonemes and/or full words, has at
least two advantages: (i) it provides a unique vocabulary representation, where
each token has a semantic meaning, in contrast to the phoneme-level
representations, (ii) each subword token has a reusable "sub"-word acoustic
pattern (that can be used to construct multiple full words), resulting in a
largely reduced vocabulary space than of the full words. To learn the subword
representations for the audio-to-intent classification, we extract: (i)
acoustic information from an E2E-ASR model, which provides frame-level CTC
posterior probabilities for the subword tokens, and (ii) textual information
from a pre-trained continuous bag-of-words model capturing the semantic meaning
of the subword tokens. The key to our approach is the way it combines acoustic
subword-level posteriors with text information using the notion of
positional-encoding in order to account for multiple ASR hypotheses
simultaneously. We show that our approach provides more robust and richer
representations for audio-to-intent classification, and is highly accurate with
correctly mitigating 93.3% of unintended user audio from invoking the smart
assistant at 99% true positive rate.
