  We consider deep feedforward neural networks with rectified linear units from
a signal processing perspective. In this view, such representations mark the
transition from using a single (data-driven) linear representation to utilizing
a large collection of affine linear representations tailored to particular
regions of the signal space. This paper provides a precise description of the
individual affine linear representations and corresponding domain regions that
the (data-driven) neural network associates to each signal of the input space.
In particular, we describe atomic decompositions of the representations and,
based on estimating their Lipschitz regularity, suggest some conditions that
can stabilize learning independent of the network depth. Such an analysis may
promote further theoretical insight from both the signal processing and machine
learning communities.
