  The computation necessary for training Transformer-based language models has
skyrocketed in recent years. This trend has motivated research on efficient
training algorithms designed to improve training, validation, and downstream
performance faster than standard training. In this work, we revisit three
categories of such algorithms: dynamic architectures (layer stacking, layer
dropping), batch selection (selective backprop, RHO loss), and efficient
optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed
computation budget using such methods, we find that their training, validation,
and downstream gains vanish compared to a baseline with a fully-decayed
learning rate. We define an evaluation protocol that enables computation to be
done on arbitrary machines by mapping all computation time to a reference
machine which we call reference system time. We discuss the limitations of our
proposed protocol and release our code to encourage rigorous research in
efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.
