  Many machine translation toolkits make use of a data preparation step wherein
raw data is transformed into a tensor format that can be used directly by the
trainer. This preparation step is increasingly at odds with modern research and
development practices because this process produces a static, unchangeable
version of the training data, making common training-time needs difficult
(e.g., subword sampling), time-consuming (preprocessing with large data can
take days), expensive (e.g., disk space), and cumbersome (managing experiment
combinatorics). We propose an alternative approach that separates the
generation of data from the consumption of that data. In this approach, there
is no separate pre-processing step; data generation produces an infinite stream
of permutations of the raw training data, which the trainer tensorizes and
batches as it is consumed. Additionally, this data stream can be manipulated by
a set of user-definable operators that provide on-the-fly modifications, such
as data normalization, augmentation or filtering. We release an open-source
toolkit, SOTASTREAM, that implements this approach:
https://github.com/marian-nmt/sotastream. We show that it cuts training time,
adds flexibility, reduces experiment management complexity, and reduces disk
space, all without affecting the accuracy of the trained models.
