  We study the role of an essential hyper-parameter that governs the training
of Transformers for neural machine translation in a low-resource setting: the
batch size. Using theoretical insights and experimental evidence, we argue
against the widespread belief that batch size should be set as large as allowed
by the memory of the GPUs. We show that in a low-resource setting, a smaller
batch size leads to higher scores in a shorter training time, and argue that
this is due to better regularization of the gradients during training.
