  The utility of a learned neural representation depends on how well its
geometry supports performance in downstream tasks. This geometry depends on the
structure of the inputs, the structure of the target outputs, and the
architecture of the network. By studying the learning dynamics of networks with
one hidden layer, we discovered that the network's activation function has an
unexpectedly strong impact on the representational geometry: Tanh networks tend
to learn representations that reflect the structure of the target outputs,
while ReLU networks retain more information about the structure of the raw
inputs. This difference is consistently observed across a broad class of
parameterized tasks in which we modulated the degree of alignment between the
geometry of the task inputs and that of the task labels. We analyzed the
learning dynamics in weight space and show how the differences between the
networks with Tanh and ReLU nonlinearities arise from the asymmetric asymptotic
behavior of ReLU, which leads feature neurons to specialize for different
regions of input space. By contrast, feature neurons in Tanh networks tend to
inherit the task label structure. Consequently, when the target outputs are low
dimensional, Tanh networks generate neural representations that are more
disentangled than those obtained with a ReLU nonlinearity. Our findings shed
light on the interplay between input-output geometry, nonlinearity, and learned
representations in neural networks.
