  Many feedforward neural networks (NNs) generate continuous and
piecewise-linear (CPWL) mappings. Specifically, they partition the input domain
into regions on which the mapping is affine. The number of these so-called
linear regions offers a natural metric to characterize the expressiveness of
CPWL NNs. The precise determination of this quantity is often out of reach in
practice, and bounds have been proposed for specific architectures, including
for ReLU and Maxout NNs. In this work, we generalize these bounds to NNs with
arbitrary and possibly multivariate CPWL activation functions. We first provide
upper and lower bounds on the maximal number of linear regions of a CPWL NN
given its depth, width, and the number of linear regions of its activation
functions. Our results rely on the combinatorial structure of convex partitions
and confirm the distinctive role of depth which, on its own, is able to
exponentially increase the number of regions. We then introduce a complementary
stochastic framework to estimate the average number of linear regions produced
by a CPWL NN. Under reasonable assumptions, the expected density of linear
regions along any 1D path is bounded by the product of depth, width, and a
measure of activation complexity (up to a scaling factor). This yields an
identical role to the three sources of expressiveness: no exponential growth
with depth is observed anymore.
