  Memory Networks have emerged as effective models to incorporate Knowledge
Bases (KB) into neural networks. By storing KB embeddings into a memory
component, these models can learn meaningful representations that are grounded
to external knowledge. However, as the memory unit becomes full, the oldest
memories are replaced by newer representations.
  In this paper, we question this approach and provide experimental evidence
that conventional Memory Networks store highly correlated vectors during
training. While increasing the memory size mitigates this problem, this also
leads to overfitting as the memory stores a large number of training latent
representations. To address these issues, we propose a novel regularization
mechanism named memory dropout which 1) Samples a single latent vector from the
distribution of redundant memories. 2) Ages redundant memories thus increasing
their probability of overwriting them during training. This fully
differentiable technique allows us to achieve state-of-the-art response
generation in the Stanford Multi-Turn Dialogue and Cambridge Restaurant
datasets.
