  In this work, we investigate the implicit regularization induced by
teacher-student learning dynamics in self-distillation. To isolate its effect,
we describe a simple experiment where we consider teachers at random
initialization instead of trained teachers. Surprisingly, when distilling a
student into such a random teacher, we observe that the resulting model and its
representations already possess very interesting characteristics; (1) we
observe a strong improvement of the distilled student over its teacher in terms
of probing accuracy. (2) The learned representations are data-dependent and
transferable between different tasks but deteriorate strongly if trained on
random inputs. (3) The student checkpoint contains sparse subnetworks,
so-called lottery tickets, and lies on the border of linear basins in the
supervised loss landscape. These observations have interesting consequences for
several important areas in machine learning: (1) Self-distillation can work
solely based on the implicit regularization present in the gradient dynamics
without relying on any dark knowledge, (2) self-supervised learning can learn
features even in the absence of data augmentation and (3) training dynamics
during the early phase of supervised training do not necessarily require label
information. Finally, we shed light on an intriguing local property of the loss
landscape: the process of feature learning is strongly amplified if the student
is initialized closely to the teacher. These results raise interesting
questions about the nature of the landscape that have remained unexplored so
far. Code is available at https://github.com/safelix/dinopl.
