  We study data distillation for auto-regressive machine learning tasks, where
the input and output have a strict left-to-right causal structure. More
specifically, we propose Farzi, which summarizes an event sequence dataset into
a small number of synthetic sequences -- Farzi Data -- which are optimized to
maintain (if not improve) model performance compared to training on the full
dataset. Under the hood, Farzi conducts memory-efficient data distillation by
(i) deriving efficient reverse-mode differentiation of the Adam optimizer by
leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
discrete event-space into a latent-space which provably promotes implicit
regularization. Empirically, for sequential recommendation and language
modeling tasks, we are able to achieve 98-120% of downstream full-data
performance when training state-of-the-art models on Farzi Data of size as
little as 0.1% of the original dataset. Notably, being able to train better
models with significantly less data sheds light on the design of future large
auto-regressive models, and opens up new opportunities to further scale up
model and data sizes.
