  In recent years, many deep neural architectures have been developed for image
classification. Whether they are similar or dissimilar and what factors
contribute to their (dis)similarities remains curious. To address this
question, we aim to design a quantitative and scalable similarity measure
between neural architectures. We propose Similarity by Attack Transferability
(SAT) from the observation that adversarial attack transferability contains
information related to input gradients and decision boundaries widely used to
understand model behaviors. We conduct a large-scale analysis on 69
state-of-the-art ImageNet classifiers using our proposed similarity function to
answer the question. Moreover, we observe neural architecture-related phenomena
using model similarity that model diversity can lead to better performance on
model ensembles and knowledge distillation under specific conditions. Our
results provide insights into why developing diverse neural architectures with
distinct components is necessary.
