  Traveling waves are a fundamental phenomenon in the brain, playing a crucial
role in short-term information storage. In this study, we leverage the concept
of traveling wave dynamics within a neural lattice to formulate a theoretical
model of neural working memory, study its properties, and its real world
implications in AI. The proposed model diverges from traditional approaches,
which assume information storage in static, register-like locations updated by
interference. Instead, the model stores data as waves that is updated by the
wave's boundary conditions. We rigorously examine the model's capabilities in
representing and learning state histories, which are vital for learning
history-dependent dynamical systems. The findings reveal that the model
reliably stores external information and enhances the learning process by
addressing the diminishing gradient problem. To understand the model's
real-world applicability, we explore two cases: linear boundary condition (LBC)
and non-linear, self-attention-driven boundary condition (SBC). The model with
the linear boundary condition results in a shift matrix plus low-rank matrix
currently used in H3 state space RNN. Further, our experiments with LBC reveal
that this matrix is effectively learned by Recurrent Neural Networks (RNNs)
through backpropagation when modeling history-dependent dynamical systems.
Conversely, the SBC parallels the autoregressive loop of an attention-only
transformer with the context vector representing the wave substrate.
Collectively, our findings suggest the broader relevance of traveling waves in
AI and its potential in advancing neural network architectures.
