  This paper presents a phenomenon in neural networks that we refer to as
\textit{local elasticity}. Roughly speaking, a classifier is said to be locally
elastic if its prediction at a feature vector $\bx'$ is \textit{not}
significantly perturbed, after the classifier is updated via stochastic
gradient descent at a (labeled) feature vector $\bx$ that is
\textit{dissimilar} to $\bx'$ in a certain sense. This phenomenon is shown to
persist for neural networks with nonlinear activation functions through
extensive simulations on real-life and synthetic datasets, whereas this is not
observed in linear classifiers. In addition, we offer a geometric
interpretation of local elasticity using the neural tangent kernel
\citep{jacot2018neural}. Building on top of local elasticity, we obtain
pairwise similarity measures between feature vectors, which can be used for
clustering in conjunction with $K$-means. The effectiveness of the clustering
algorithm on the MNIST and CIFAR-10 datasets in turn corroborates the
hypothesis of local elasticity of neural networks on real-life data. Finally,
we discuss some implications of local elasticity to shed light on several
intriguing aspects of deep neural networks.
