  This paper presents a comprehensive analysis of critical point sets in
two-layer neural networks. To study such complex entities, we introduce the
critical embedding operator and critical reduction operator as our tools. Given
a critical point, we use these operators to uncover the whole underlying
critical set representing the same output function, which exhibits a
hierarchical structure. Furthermore, we prove existence of saddle branches for
any critical set whose output function can be represented by a narrower
network. Our results provide a solid foundation to the further study of
optimization and training behavior of neural networks.
