  The computational complexity of the self-attention mechanism in Transformer
models significantly limits their ability to generalize over long temporal
durations. Memory-augmentation, or the explicit storing of past information in
external memory for subsequent predictions, has become a constructive avenue
for mitigating this limitation. We argue that memory-augmented Transformers can
benefit substantially from considering insights from the memory literature in
humans. We detail an approach for integrating evidence from the human memory
system through the specification of cross-domain linking hypotheses. We then
provide an empirical demonstration to evaluate the use of surprisal as a
linking hypothesis, and further identify the limitations of this approach to
inform future research.
