  Spoken dialogue systems such as Siri and Alexa provide great convenience to
people's everyday life. However, current spoken language understanding (SLU)
pipelines largely depend on automatic speech recognition (ASR) modules, which
require a large amount of language-specific training data. In this paper, we
propose a Transformer-based SLU system that works directly on phones. This
acoustic-based SLU system consists of only two blocks and does not require the
presence of ASR module. The first block is a universal phone recognition
system, and the second block is a Transformer-based language model for phones.
We verify the effectiveness of the system on an intent classification dataset
in Mandarin Chinese.
