  A characterization of the representability of neural networks is relevant to
comprehend their success in artificial intelligence. This study investigate two
topics on ReLU neural network expressivity and their connection with a
conjecture related to the minimum depth required for representing any
continuous piecewise linear (CPWL) function. The topics are the minimal depth
representation of the sum and max operations, as well as the exploration of
polytope neural networks. For the sum operation, we establish a sufficient
condition on the minimal depth of the operands to find the minimal depth of the
operation. In contrast, regarding the max operation, a comprehensive set of
examples is presented, demonstrating that no sufficient conditions, depending
solely on the depth of the operands, would imply a minimal depth for the
operation. The study also examine the minimal depth relationship between convex
CPWL functions. On polytope neural networks, we investigate basic depth
properties from Minkowski sums, convex hulls, number of vertices, faces, affine
transformations, and indecomposable polytopes. More significant findings
include depth characterization of polygons; identification of polytopes with an
increasing number of vertices, exhibiting small depth and others with arbitrary
large depth; and most notably, the minimal depth of simplices, which is
strictly related to the minimal depth conjecture in ReLU networks.
