  In this paper, the bias classifier is introduced, that is, the bias part of a
DNN with Relu as the activation function is used as a classifier. The work is
motivated by the fact that the bias part is a piecewise constant function with
zero gradient and hence cannot be directly attacked by gradient-based methods
to generate adversaries, such as FGSM. The existence of the bias classifier is
proved and an effective training method for the bias classifier is given. It is
proved that by adding a proper random first-degree part to the bias classifier,
an information-theoretically safe classifier against the original-model
gradient attack is obtained in the sense that the attack will generate a
totally random attacking direction. This seems to be the first time that the
concept of information-theoretically safe classifier is proposed. Several
attack methods for the bias classifier are proposed and numerical experiments
are used to show that the bias classifier is more robust than DNNs with similar
size against these attacks in most cases.
