  Understanding the intricate operations of Recurrent Neural Networks (RNNs)
mechanistically is pivotal for advancing their capabilities and applications.
In this pursuit, we propose the Episodic Memory Theory (EMT), illustrating that
RNNs can be conceptualized as discrete-time analogs of the recently proposed
General Sequential Episodic Memory Model. To substantiate EMT, we introduce a
novel set of algorithmic tasks tailored to probe the variable binding behavior
in RNNs. Utilizing the EMT, we formulate a mathematically rigorous circuit that
facilitates variable binding in these tasks. Our empirical investigations
reveal that trained RNNs consistently converge to the variable binding circuit,
thus indicating universality in the dynamics of RNNs. Building on these
findings, we devise an algorithm to define a privileged basis, which reveals
hidden neurons instrumental in the temporal storage and composition of
variables, a mechanism vital for the successful generalization in these tasks.
We show that the privileged basis enhances the interpretability of the learned
parameters and hidden states of RNNs. Our work represents a step toward
demystifying the internal mechanisms of RNNs and, for computational
neuroscience, serves to bridge the gap between artificial neural networks and
neural memory models.
