  The universal approximation property is fundamental to the success of neural
networks, and has traditionally been achieved by training networks without any
constraints on their parameters. However, recent experimental research proposed
a novel permutation-based training method, which exhibited a desired
classification performance without modifying the exact weight values. In this
paper, we provide a theoretical guarantee of this permutation training method
by proving its ability to guide a ReLU network to approximate one-dimensional
continuous functions. Our numerical results further validate this method's
efficiency in regression tasks with various initializations. The notable
observations during weight permutation suggest that permutation training can
provide an innovative tool for describing network learning behavior.
