  A deep neural network using rectified linear units represents a continuous
piecewise linear (CPWL) function and vice versa. Recent results in the
literature estimated that the number of neurons needed to exactly represent any
CPWL function grows exponentially with the number of pieces or exponentially in
terms of the factorial of the number of distinct linear components. Moreover,
such growth is amplified linearly with the input dimension. These existing
results seem to indicate that the cost of representing a CPWL function is
expensive. In this paper, we propose much tighter bounds and establish a
polynomial time algorithm to find a network satisfying these bounds for any
given CPWL function. We prove that the number of hidden neurons required to
exactly represent any CPWL function is at most a quadratic function of the
number of pieces. In contrast to all previous results, this upper bound is
invariant to the input dimension. Besides the number of pieces, we also study
the number of distinct linear components in CPWL functions. When such a number
is also given, we prove that the quadratic complexity turns into bilinear,
which implies a lower neural complexity because the number of distinct linear
components is always not greater than the minimum number of pieces in a CPWL
function. When the number of pieces is unknown, we prove that, in terms of the
number of distinct linear components, the neural complexities of any CPWL
function are at most polynomial growth for low-dimensional inputs and factorial
growth for the worst-case scenario, which are significantly better than
existing results in the literature.
