  Neural networks are commonly used as models for classification for a wide
variety of tasks. Typically, a learned affine transformation is placed at the
end of such models, yielding a per-class value used for classification. This
classifier can have a vast number of parameters, which grows linearly with the
number of possible classes, thus requiring increasingly more resources. In this
work we argue that this classifier can be fixed, up to a global scale constant,
with little or no loss of accuracy for most tasks, allowing memory and
computational benefits. Moreover, we show that by initializing the classifier
with a Hadamard matrix we can speed up inference as well. We discuss the
implications for current understanding of neural network models.
