  Training large-scale image recognition models is computationally expensive.
This raises the question of whether there might be simple ways to improve the
test performance of an already trained model without having to re-train or
fine-tune it with new data. Here, we show that, surprisingly, this is indeed
possible. The key observation we make is that the layers of a deep network
close to the output layer contain independent, easily extractable
class-relevant information that is not contained in the output layer itself. We
propose to extract this extra class-relevant information using a simple
key-value cache memory to improve the classification performance of the model
at test time. Our cache memory is directly inspired by a similar cache model
previously proposed for language modeling (Grave et al., 2017). This cache
component does not require any training or fine-tuning; it can be applied to
any pre-trained model and, by properly setting only two hyper-parameters, leads
to significant improvements in its classification performance. Improvements are
observed across several architectures and datasets. In the cache component,
using features extracted from layers close to the output (but not from the
output layer itself) as keys leads to the largest improvements. Concatenating
features from multiple layers to form keys can further improve performance over
using single-layer features as keys. The cache component also has a
regularizing effect, a simple consequence of which is that it substantially
increases the robustness of models against adversarial attacks.
