  A mechanistic understanding of the computations learned by deep neural
networks (DNNs) is far from complete. In the domain of visual object
recognition, prior research has illuminated inner workings of InceptionV1, but
DNNs with different architectures have remained largely unexplored. This work
investigates ResNet18 with a particular focus on its residual stream, an
architectural mechanism which InceptionV1 lacks. We observe that for a given
block, channel features of the stream are updated along a spectrum: either the
input feature skips to the output, the block feature overwrites the output, or
the output is some mixture between the input and block features. Furthermore,
we show that many residual stream channels compute scale invariant
representations through a mixture of the input's smaller-scale feature with the
block's larger-scale feature. This not only mounts evidence for the
universality of scale equivariance, but also presents how the residual stream
further implements scale invariance. Collectively, our results begin an
interpretation of the residual stream in visual object recognition, finding it
to be a flexible feature manager and a medium to build scale invariant
representations.
