  After just a few hundred training updates, a standard probabilistic model for
language generation has likely not yet learnt many semantic or syntactic rules
of natural language, making it difficult to estimate the probability
distribution over next tokens. Yet around this point, these models have
identified a simple, loss-minimising behaviour: to output the unigram
distribution of the target training corpus. The use of such a heuristic raises
the question: Can we initialise our models with this behaviour and save
precious compute resources and model capacity? Here we show that we can
effectively endow standard neural language generation models with a separate
module that reflects unigram frequency statistics as prior knowledge, simply by
initialising the bias term in a model's final linear layer with the log-unigram
distribution. We use neural machine translation as a test bed for this simple
technique and observe that it: (i) improves learning efficiency; (ii) achieves
better overall performance; and perhaps most importantly (iii) appears to
disentangle strong frequency effects by encouraging the model to specialise in
non-frequency-related aspects of language.
