  Model multiplicity is a well-known but poorly understood phenomenon that
undermines the generalisation guarantees of machine learning models. It appears
when two models with similar training-time performance differ in their
predictions and real-world performance characteristics. This observed
'predictive' multiplicity (PM) also implies elusive differences in the
internals of the models, their 'representational' multiplicity (RM). We
introduce a conceptual and experimental setup for analysing RM by measuring
activation similarity via singular vector canonical correlation analysis
(SVCCA). We show that certain differences in training methods systematically
result in larger RM than others and evaluate RM and PM over a finite sample as
predictors for generalizability. We further correlate RM with PM measured by
the variance in i.i.d. and out-of-distribution test predictions in four
standard image data sets. Finally, instead of attempting to eliminate RM, we
call for its systematic measurement and maximal exposure.
