  Transformer-based neural networks have achieved state-of-the-art task
performance in a number of machine learning domains including natural language
processing and computer vision. To further improve their accuracy, recent work
has explored the integration of dynamic behavior into these networks in the
form of mixture-of-expert (MoE) layers. In this paper, we explore the
introduction of MoE layers to optimize a different metric: inference latency.
We introduce a novel system named PLANER that takes an existing
Transformer-based network and a user-defined latency target and produces an
optimized, sparsely-activated version of the original network that tries to
meet the latency target while maintaining baseline accuracy. We evaluate PLANER
on two real-world language modeling tasks using the Transformer-XL network and
achieve inference latency reductions of over 2x at iso-accuracy.
