  Batch normalization (BN) has been widely used in modern deep neural networks
(DNNs) due to improved convergence. BN is observed to increase the model
accuracy while at the cost of adversarial robustness. There is an increasing
interest in the ML community to understand the impact of BN on DNNs, especially
related to the model robustness. This work attempts to understand the impact of
BN on DNNs from a non-robust feature perspective. Straightforwardly, the
improved accuracy can be attributed to the better utilization of useful
features. It remains unclear whether BN mainly favors learning robust features
(RFs) or non-robust features (NRFs). Our work presents empirical evidence that
supports that BN shifts a model towards being more dependent on NRFs. To
facilitate the analysis of such a feature robustness shift, we propose a
framework for disentangling robust usefulness into robustness and usefulness.
Extensive analysis under the proposed framework yields valuable insight on the
DNN behavior regarding robustness, e.g. DNNs first mainly learn RFs and then
NRFs. The insight that RFs transfer better than NRFs, further inspires simple
techniques to strengthen transfer-based black-box attacks.
