  The impressive generalization performance of modern neural networks is
attributed in part to their ability to implicitly memorize complex training
patterns. Inspired by this, we explore a novel mechanism to improve model
generalization via explicit memorization. Specifically, we propose the
residual-memorization (ResMem) algorithm, a new method that augments an
existing prediction model (e.g. a neural network) by fitting the model's
residuals with a $k$-nearest neighbor based regressor. The final prediction is
then the sum of the original model and the fitted residual regressor. By
construction, ResMem can explicitly memorize the training labels. Empirically,
we show that ResMem consistently improves the test set generalization of the
original prediction model across various standard vision and natural language
processing benchmarks. Theoretically, we formulate a stylized linear regression
problem and rigorously show that ResMem results in a more favorable test risk
over the base predictor.
