  Recent studies suggest that deep learning models inductive bias towards
favoring simpler features may be one of the sources of shortcut learning. Yet,
there has been limited focus on understanding the complexity of the myriad
features that models learn. In this work, we introduce a new metric for
quantifying feature complexity, based on $\mathscr{V}$-information and
capturing whether a feature requires complex computational transformations to
be extracted. Using this $\mathscr{V}$-information metric, we analyze the
complexities of 10,000 features, represented as directions in the penultimate
layer, that were extracted from a standard ImageNet-trained vision model. Our
study addresses four key questions: First, we ask what features look like as a
function of complexity and find a spectrum of simple to complex features
present within the model. Second, we ask when features are learned during
training. We find that simpler features dominate early in training, and more
complex features emerge gradually. Third, we investigate where within the
network simple and complex features flow, and find that simpler features tend
to bypass the visual hierarchy via residual connections. Fourth, we explore the
connection between features complexity and their importance in driving the
networks decision. We find that complex features tend to be less important.
Surprisingly, important features become accessible at earlier layers during
training, like a sedimentation process, allowing the model to build upon these
foundational elements.
