  Robustness against adversarial attack in neural networks is an important
research topic in the machine learning community. We observe one major source
of vulnerability of neural nets is from overparameterized fully-connected
layers. In this paper, we propose a new neighborhood preserving layer which can
replace these fully connected layers to improve the network robustness. We
demonstrate a novel neural network architecture which can incorporate such
layers and also can be trained efficiently. We theoretically prove that our
models are more robust against distortion because they effectively control the
magnitude of gradients. Finally, we empirically show that our designed network
architecture is more robust against state-of-art gradient descent based
attacks, such as a PGD attack on the benchmark datasets MNIST and CIFAR10.
