  The rapid growth of neural network models shared on the internet has made
model weights an important data modality. However, this information is
underutilized as the weights are uninterpretable, and publicly available models
are disorganized. Inspired by Darwin's tree of life, we define the Model Tree
which describes the origin of models i.e., the parent model that was used to
fine-tune the target model. Similarly to the natural world, the tree structure
is unknown. In this paper, we introduce the task of Model Tree Heritage
Recovery (MoTHer Recovery) for discovering Model Trees in the ever-growing
universe of neural networks. Our hypothesis is that model weights encode this
information, the challenge is to decode the underlying tree structure given the
weights. Beyond the immediate application of model authorship attribution,
MoTHer recovery holds exciting long-term applications akin to indexing the
internet by search engines. Practically, for each pair of models, this task
requires: i) determining if they are related, and ii) establishing the
direction of the relationship. We find that certain distributional properties
of the weights evolve monotonically during training, which enables us to
classify the relationship between two given models. MoTHer recovery
reconstructs entire model hierarchies, represented by a directed tree, where a
parent model gives rise to multiple child models through additional training.
Our approach successfully reconstructs complex Model Trees, as well as the
structure of "in-the-wild" model families such as Llama 2 and Stable Diffusion.
