  Exploring the loss landscape offers insights into the inherent principles of
deep neural networks (DNNs). Recent work suggests an additional asymmetry of
the valley beyond the flat and sharp ones, yet without thoroughly examining its
causes or implications. Our study methodically explores the factors affecting
the symmetry of DNN valleys, encompassing (1) the dataset, network
architecture, initialization, and hyperparameters that influence the
convergence point; and (2) the magnitude and direction of the noise for 1D
visualization. Our major observation shows that the {\it degree of sign
consistency} between the noise and the convergence point is a critical
indicator of valley symmetry. Theoretical insights from the aspects of ReLU
activation and softmax function could explain the interesting phenomenon. Our
discovery propels novel understanding and applications in the scenario of Model
Fusion: (1) the efficacy of interpolating separate models significantly
correlates with their sign consistency ratio, and (2) imposing sign alignment
during federated learning emerges as an innovative approach for model parameter
alignment.
