  It has been well established that increasing scale in deep transformer
networks leads to improved quality and performance. However, this increase in
scale often comes with prohibitive increases in compute cost and inference
latency. We introduce Alternating Updates (AltUp), a simple-to-implement method
to increase a model's capacity without the computational burden. AltUp enables
the widening of the learned representation, i.e., the token embedding, while
only incurring a negligible increase in latency. AltUp achieves this by working
on a subblock of the widened representation at each layer and using a
predict-and-correct mechanism to update the inactivated blocks. We present
extensions of AltUp, such as its applicability to the sequence dimension, and
demonstrate how AltUp can be synergistically combined with existing approaches,
such as Sparse Mixture-of-Experts models, to obtain efficient models with even
higher capacity. Our experiments on benchmark transformer models and language
tasks demonstrate the consistent effectiveness of AltUp on a diverse set of
scenarios. Notably, on SuperGLUE and SQuAD benchmarks, AltUp enables up to
$87\%$ speedup relative to the dense baselines at the same accuracy.
