  Neural Collapse refers to the curious phenomenon in the end of training of a
neural network, where feature vectors and classification weights converge to a
very simple geometrical arrangement (a simplex). While it has been observed
empirically in various cases and has been theoretically motivated, its
connection with crucial properties of neural networks, like their
generalization and robustness, remains unclear. In this work, we study the
stability properties of these simplices. We find that the simplex structure
disappears under small adversarial attacks, and that perturbed examples "leap"
between simplex vertices. We further analyze the geometry of networks that are
optimized to be robust against adversarial perturbations of the input, and find
that Neural Collapse is a pervasive phenomenon in these cases as well, with
clean and perturbed representations forming aligned simplices, and giving rise
to a robust simple nearest-neighbor classifier. By studying the propagation of
the amount of collapse inside the network, we identify novel properties of both
robust and non-robust machine learning models, and show that earlier, unlike
later layers maintain reliable simplices on perturbed data.
