  Comparing different neural network representations and determining how
representations evolve over time remain challenging open questions in our
understanding of the function of neural networks. Comparing representations in
neural networks is fundamentally difficult as the structure of representations
varies greatly, even across groups of networks trained on identical tasks, and
over the course of training. Here, we develop projection weighted CCA
(Canonical Correlation Analysis) as a tool for understanding neural networks,
building off of SVCCA, a recently proposed method (Raghu et al., 2017). We
first improve the core method, showing how to differentiate between signal and
noise, and then apply this technique to compare across a group of CNNs,
demonstrating that networks which generalize converge to more similar
representations than networks which memorize, that wider networks converge to
more similar solutions than narrow networks, and that trained networks with
identical topology but different learning rates converge to distinct clusters
with diverse representations. We also investigate the representational dynamics
of RNNs, across both training and sequential timesteps, finding that RNNs
converge in a bottom-up pattern over the course of training and that the hidden
state is highly variable over the course of a sequence, even when accounting
for linear transforms. Together, these results provide new insights into the
function of CNNs and RNNs, and demonstrate the utility of using CCA to
understand representations.
