  The outstanding accuracy achieved by modern Automatic Speech Recognition
(ASR) systems is enabling them to quickly become a mainstream technology. ASR
is essential for many applications, such as speech-based assistants, dictation
systems and real-time language translation. However, highly accurate ASR
systems are computationally expensive, requiring on the order of billions of
arithmetic operations to decode each second of audio, which conflicts with a
growing interest in deploying ASR on edge devices. On these devices, hardware
acceleration is key for achieving acceptable performance. However, ASR is a
rich and fast-changing field, and thus, any overly specialized hardware
accelerator may quickly become obsolete.
  In this paper, we tackle those challenges by proposing ASRPU, a programmable
accelerator for on-edge ASR. ASRPU contains a pool of general-purpose cores
that execute small pieces of parallel code. Each of these programs computes one
part of the overall decoder (e.g. a layer in a neural network). The accelerator
automates some carefully chosen parts of the decoder to simplify the
programming without sacrificing generality. We provide an analysis of a modern
ASR system implemented on ASRPU and show that this architecture can achieve
real-time decoding with a very low power budget.
