  Data, the seminal opportunity and challenge in modern machine learning,
currently constrains the scalability of representation learning and impedes the
pace of model evolution. Existing paradigms tackle the issue of learning
efficiency over massive datasets from the perspective of self-supervised
learning and dataset distillation independently, while neglecting the untapped
potential of accelerating representation learning from an intermediate
standpoint. In this work, we delve into defining the ideal data properties from
both optimization and generalization perspectives. We propose that
model-generated representations, despite being trained on diverse tasks and
architectures, converge to a shared linear space, facilitating effective linear
transport between models. Furthermore, we demonstrate that these
representations exhibit properties conducive to the formation of ideal data.
The theoretical/empirical insights therein inspire us to propose a
Representation Learning Accelerator (ReLA), which leverages a task- and
architecture-agnostic, yet publicly available, free model to form a dynamic
data subset and thus accelerate (self-)supervised learning. For instance,
employing a CLIP ViT B/16 as a prior model for dynamic data generation,
ReLA-aided BYOL can train a ResNet-50 from scratch with 50% of ImageNet-1K,
yielding performance surpassing that of training on the full dataset.
Additionally, employing a ResNet-18 pre-trained on CIFAR-10 can enhance
ResNet-50 training on 10% of ImageNet-1K, resulting in a 7.7% increase in
accuracy.
