  It is commonly recognized that the expressiveness of deep neural networks is
contingent upon a range of factors, encompassing their depth, width, and other
relevant considerations. Currently, the practical performance of the majority
of deep neural networks remains uncertain. For ReLU (Rectified Linear Unit)
networks with piecewise linear activations, the number of linear convex regions
serves as a natural metric to gauge the network's expressivity. In this paper,
we count the number of linear convex regions in deep neural networks based on
ReLU. In particular, we prove that for any one-dimensional input, there exists
a minimum threshold for the number of neurons required to express it. We also
empirically observe that for the same network, intricate inputs hinder its
capacity to express linear regions. Furthermore, we unveil the iterative
refinement process of decision boundaries in ReLU networks during training. We
aspire for our research to serve as an inspiration for network optimization
endeavors and aids in the exploration and analysis of the behaviors exhibited
by deep networks.
