  It is often said that a deep learning model is "invariant" to some specific
type of transformation. However, what is meant by this statement strongly
depends on the context in which it is made. In this paper we explore the nature
of invariance and equivariance of deep learning models with the goal of better
understanding the ways in which they actually capture these concepts on a
formal level. We introduce a family of invariance and equivariance metrics that
allows us to quantify these properties in a way that disentangles them from
other metrics such as loss or accuracy. We use our metrics to better understand
the two most popular methods used to build invariance into networks: data
augmentation and equivariant layers. We draw a range of conclusions about
invariance and equivariance in deep learning models, ranging from whether
initializing a model with pretrained weights has an effect on a trained model's
invariance, to the extent to which invariance learned via training can
generalize to out-of-distribution data.
