  The developments of deep neural networks (DNN) in recent years have ushered a
brand new era of artificial intelligence. DNNs are proved to be excellent in
solving very complex problems, e.g., visual recognition and text understanding,
to the extent of competing with or even surpassing people. Despite inspiring
and encouraging success of DNNs, thorough theoretical analyses still lack to
unravel the mystery of their magics. The design of DNN structure is dominated
by empirical results in terms of network depth, number of neurons and
activations. A few of remarkable works published recently in an attempt to
interpret DNNs have established the first glimpses of their internal
mechanisms. Nevertheless, research on exploring how DNNs operate is still at
the initial stage with plenty of room for refinement. In this paper, we extend
precedent research on neural networks with piecewise linear activations (PLNN)
concerning linear regions bounds. We present (i) the exact maximal number of
linear regions for single layer PLNNs; (ii) a upper bound for multi-layer
PLNNs; and (iii) a tighter upper bound for the maximal number of liner regions
on rectifier networks. The derived bounds also indirectly explain why deep
models are more powerful than shallow counterparts, and how non-linearity of
activation functions impacts on expressiveness of networks.
