  Recently, a weight-based attack on stochastic gradient descent inducing
overfitting has been proposed. We show that the threat is broader: A
task-independent permutation on the initial weights suffices to limit the
achieved accuracy to for example 50% on the Fashion MNIST dataset from
initially more than $90$%. These findings are confirmed on MNIST and CIFAR. We
formally confirm that the attack succeeds with high likelihood and does not
depend on the data. Empirically, weight statistics and loss appear
unsuspicious, making it hard to detect the attack if the user is not aware. Our
paper is thus a call for action to acknowledge the importance of the initial
weights in deep learning.
