  Deep neural networks have become an integral part of our software
infrastructure and are being deployed in many widely-used and safety-critical
applications. However, their integration into many systems also brings with it
the vulnerability to test time attacks in the form of Universal Adversarial
Perturbations (UAPs). UAPs are a class of perturbations that when applied to
any input causes model misclassification. Although there is an ongoing effort
to defend models against these adversarial attacks, it is often difficult to
reconcile the trade-offs in model accuracy and robustness to adversarial
attacks. Jacobian regularization has been shown to improve the robustness of
models against UAPs, whilst model ensembles have been widely adopted to improve
both predictive performance and model robustness. In this work, we propose a
novel approach, Jacobian Ensembles-a combination of Jacobian regularization and
model ensembles to significantly increase the robustness against UAPs whilst
maintaining or improving model accuracy. Our results show that Jacobian
Ensembles achieves previously unseen levels of accuracy and robustness, greatly
improving over previous methods that tend to skew towards only either accuracy
or robustness.
