  Deep learning has become an integral part of various computer vision systems
in recent years due to its outstanding achievements for object recognition,
facial recognition, and scene understanding. However, deep neural networks
(DNNs) are susceptible to be fooled with nearly high confidence by an
adversary. In practice, the vulnerability of deep learning systems against
carefully perturbed images, known as adversarial examples, poses a dire
security threat in the physical world applications. To address this phenomenon,
we present, what to our knowledge, is the first ever image set based
adversarial defence approach. Image set classification has shown an exceptional
performance for object and face recognition, owing to its intrinsic property of
handling appearance variability. We propose a robust deep Bayesian image set
classification as a defence framework against a broad range of adversarial
attacks. We extensively experiment the performance of the proposed technique
with several voting strategies. We further analyse the effects of image size,
perturbation magnitude, along with the ratio of perturbed images in each image
set. We also evaluate our technique with the recent state-of-the-art defence
methods, and single-shot recognition task. The empirical results demonstrate
superior performance on CIFAR-10, MNIST, ETH-80, and Tiny ImageNet datasets.
