  Deep neural network models are massively deployed on a wide variety of
hardware platforms. This results in the appearance of new attack vectors that
significantly extend the standard attack surface, extensively studied by the
adversarial machine learning community. One of the first attack that aims at
drastically dropping the performance of a model, by targeting its parameters
(weights) stored in memory, is the Bit-Flip Attack (BFA). In this work, we
point out several evaluation challenges related to the BFA. First of all, the
lack of an adversary's budget in the standard threat model is problematic,
especially when dealing with physical attacks. Moreover, since the BFA presents
critical variability, we discuss the influence of some training parameters and
the importance of the model architecture. This work is the first to present the
impact of the BFA against fully-connected architectures that present different
behaviors compared to convolutional neural networks. These results highlight
the importance of defining robust and sound evaluation methodologies to
properly evaluate the dangers of parameter-based attacks as well as measure the
real level of robustness offered by a defense.
