  We developed a tool for visualizing and analyzing large pre-trained vision
models by mapping them onto the brain, thus exposing their hidden inside. Our
innovation arises from a surprising usage of brain encoding: predicting brain
fMRI measurements in response to images. We report two findings. First,
explicit mapping between the brain and deep-network features across dimensions
of space, layers, scales, and channels is crucial. This mapping method,
FactorTopy, is plug-and-play for any deep-network; with it, one can paint a
picture of the network onto the brain (literally!). Second, our visualization
shows how different training methods matter: they lead to remarkable
differences in hierarchical organization and scaling behavior, growing with
more data or network capacity. It also provides insight into fine-tuning: how
pre-trained models change when adapting to small datasets. We found brain-like
hierarchically organized network suffer less from catastrophic forgetting after
fine-tuned.
