  We formalize and interpret the geometric structure of $d$-dimensional fully
connected ReLU layers in neural networks. The parameters of a ReLU layer induce
a natural partition of the input domain, such that the ReLU layer can be
significantly simplified in each sector of the partition. This leads to a
geometric interpretation of a ReLU layer as a projection onto a polyhedral cone
followed by an affine transformation, in line with the description in
[doi:10.48550/arXiv.1905.08922] for convolutional networks with ReLU
activations. Further, this structure facilitates simplified expressions for
preimages of the intersection between partition sectors and hyperplanes, which
is useful when describing decision boundaries in a classification setting. We
investigate this in detail for a feed-forward network with one hidden
ReLU-layer, where we provide results on the geometric complexity of the
decision boundary generated by such networks, as well as proving that modulo an
affine transformation, such a network can only generate $d$ different decision
boundaries. Finally, the effect of adding more layers to the network is
discussed.
