  Measuring the similarity of different representations of neural architectures
is a fundamental task and an open research challenge for the machine learning
community. This paper presents the first comprehensive benchmark for evaluating
representational similarity measures based on well-defined groundings of
similarity. The representational similarity (ReSi) benchmark consists of (i)
six carefully designed tests for similarity measures, (ii) 23 similarity
measures, (iii) eleven neural network architectures, and (iv) six datasets,
spanning over the graph, language, and vision domains. The benchmark opens up
several important avenues of research on representational similarity that
enable novel explorations and applications of neural architectures. We
demonstrate the utility of the ReSi benchmark by conducting experiments on
various neural network architectures, real world datasets and similarity
measures. All components of the benchmark are publicly available and thereby
facilitate systematic reproduction and production of research results. The
benchmark is extensible, future research can build on and further expand it. We
believe that the ReSi benchmark can serve as a sound platform catalyzing future
research that aims to systematically evaluate existing and explore novel ways
of comparing representations of neural architectures.
