  Recently, self-attention-based transformers and conformers have been
introduced as alternatives to RNNs for ASR acoustic modeling. Nevertheless, the
full-sequence attention mechanism is non-streamable and computationally
expensive, thus requiring modifications, such as chunking and caching, for
efficient streaming ASR. In this paper, we propose to apply RWKV, a variant of
linear attention transformer, to streaming ASR. RWKV combines the superior
performance of transformers and the inference efficiency of RNNs, which is
well-suited for streaming ASR scenarios where the budget for latency and memory
is restricted. Experiments on varying scales (100h - 10000h) demonstrate that
RWKV-Transducer and RWKV-Boundary-Aware-Transducer achieve comparable to or
even better accuracy compared with chunk conformer transducer, with minimal
latency and inference memory cost.
