  We prove that finding all globally optimal two-layer ReLU neural networks can
be performed by solving a convex optimization program with cone constraints.
Our analysis is novel, characterizes all optimal solutions, and does not
leverage duality-based analysis which was recently used to lift neural network
training into convex spaces. Given the set of solutions of our convex
optimization program, we show how to construct exactly the entire set of
optimal neural networks. We provide a detailed characterization of this optimal
set and its invariant transformations. As additional consequences of our convex
perspective, (i) we establish that Clarke stationary points found by stochastic
gradient descent correspond to the global optimum of a subsampled convex
problem (ii) we provide a polynomial-time algorithm for checking if a neural
network is a global minimum of the training loss (iii) we provide an explicit
construction of a continuous path between any neural network and the global
minimum of its sublevel set and (iv) characterize the minimal size of the
hidden layer so that the neural network optimization landscape has no spurious
valleys. Overall, we provide a rich framework for studying the landscape of
neural network training loss through convexity.
