  We investigate the expressivity and learning dynamics of bias-free ReLU
networks. We firstly show that two-layer bias-free ReLU networks have limited
expressivity: the only odd function two-layer bias-free ReLU networks can
express is a linear one. We then show that, under symmetry conditions on the
data, these networks have the same learning dynamics as linear networks. This
allows us to give closed-form time-course solutions to certain two-layer
bias-free ReLU networks, which has not been done for nonlinear networks outside
the lazy learning regime. While deep bias-free ReLU networks are more
expressive than their two-layer counterparts, they still share a number of
similarities with deep linear networks. These similarities enable us to
leverage insights from linear networks, leading to a novel understanding of
bias-free ReLU networks. Overall, our results show that some properties
established for bias-free ReLU networks arise due to equivalence to linear
networks, and suggest that including bias or considering asymmetric data are
avenues to engage with nonlinear behaviors.
