  Permutation symmetries of deep networks make basic operations like model
merging and similarity estimation challenging. In many cases, aligning the
weights of the networks, i.e., finding optimal permutations between their
weights, is necessary. Unfortunately, weight alignment is an NP-hard problem.
Prior research has mainly focused on solving relaxed versions of the alignment
problem, leading to either time-consuming methods or sub-optimal solutions. To
accelerate the alignment process and improve its quality, we propose a novel
framework aimed at learning to solve the weight alignment problem, which we
name Deep-Align. To that end, we first prove that weight alignment adheres to
two fundamental symmetries and then, propose a deep architecture that respects
these symmetries. Notably, our framework does not require any labeled data. We
provide a theoretical analysis of our approach and evaluate Deep-Align on
several types of network architectures and learning setups. Our experimental
results indicate that a feed-forward pass with Deep-Align produces better or
equivalent alignments compared to those produced by current optimization
algorithms. Additionally, our alignments can be used as an effective
initialization for other methods, leading to improved solutions with a
significant speedup in convergence.
