  We introduce a novel gated recurrent unit (GRU) with a weighted time-delay
feedback mechanism in order to improve the modeling of long-term dependencies
in sequential data. This model is a discretized version of a continuous-time
formulation of a recurrent unit, where the dynamics are governed by delay
differential equations (DDEs). By considering a suitable time-discretization
scheme, we propose $\tau$-GRU, a discrete-time gated recurrent unit with delay.
We prove the existence and uniqueness of solutions for the continuous-time
model, and we demonstrate that the proposed feedback mechanism can help improve
the modeling of long-term dependencies. Our empirical results show that
$\tau$-GRU can converge faster and generalize better than state-of-the-art
recurrent units and gated recurrent architectures on a range of tasks,
including time-series classification, human activity recognition, and speech
recognition.
