  We present a streaming, Transformer-based end-to-end automatic speech
recognition (ASR) architecture which achieves efficient neural inference
through compute cost amortization. Our architecture creates sparse computation
pathways dynamically at inference time, resulting in selective use of compute
resources throughout decoding, enabling significant reductions in compute with
minimal impact on accuracy. The fully differentiable architecture is trained
end-to-end with an accompanying lightweight arbitrator mechanism operating at
the frame-level to make dynamic decisions on each input while a tunable loss
function is used to regularize the overall level of compute against predictive
performance. We report empirical results from experiments using the compute
amortized Transformer-Transducer (T-T) model conducted on LibriSpeech data. Our
best model can achieve a 60% compute cost reduction with only a 3% relative
word error rate (WER) increase.
