  We introduce a neural stack architecture, including a differentiable
parametrized stack operator that approximates stack push and pop operations for
suitable choices of parameters that explicitly represents a stack. We prove the
stability of this stack architecture: after arbitrarily many stack operations,
the state of the neural stack still closely resembles the state of the discrete
stack. Using the neural stack with a recurrent neural network, we introduce a
neural network Pushdown Automaton (nnPDA) and prove that nnPDA with
finite/bounded neurons and time can simulate any PDA. Furthermore, we extend
our construction and propose new architecture neural state Turing Machine
(nnTM). We prove that differentiable nnTM with bounded neurons can simulate
Turing Machine (TM) in real-time. Just like the neural stack, these
architectures are also stable. Finally, we extend our construction to show that
differentiable nnTM is equivalent to Universal Turing Machine (UTM) and can
simulate any TM with only \textbf{seven finite/bounded precision} neurons. This
work provides a new theoretical bound for the computational capability of
bounded precision RNNs augmented with memory.
