  The storage capacity of a binary classification model is the maximum number
of random input-output pairs per parameter that the model can learn. It is one
of the indicators of the expressive power of machine learning models and is
important for comparing the performance of various models. In this study, we
analyze the structure of the solution space and the storage capacity of fully
connected two-layer neural networks with general activation functions using the
replica method from statistical physics. Our results demonstrate that the
storage capacity per parameter remains finite even with infinite width and that
the weights of the network exhibit negative correlations, leading to a
'division of labor'. In addition, we find that increasing the dataset size
triggers a phase transition at a certain transition point where the permutation
symmetry of weights is broken, resulting in the solution space splitting into
disjoint regions. We identify the dependence of this transition point and the
storage capacity on the choice of activation function. These findings
contribute to understanding the influence of activation functions and the
number of parameters on the structure of the solution space, potentially
offering insights for selecting appropriate architectures based on specific
objectives.
