  Large pretrained self-attention neural networks, or transformers, have been
very successful in various tasks recently. The performance of a model on a
given task depends on its ability to memorize and generalize the training data.
Large transformer models, which may have billions of parameters, in theory have
a huge capacity to memorize content. However, the current algorithms for the
optimization fall short of the theoretical capacity, and the capacity is also
highly dependent on the content. In this paper, we focus on the memory capacity
of these models obtained using common training algorithms and synthetic
training data. Based on the results, we derive an empirical capacity model
(ECM) for a generic transformer. The ECM can be used to design task-specific
transformer models with an optimal number of parameters in cases where the
target memorization capability of the task can be defined.
