  This paper introduces two recurrent neural network structures called Simple
Gated Unit (SGU) and Deep Simple Gated Unit (DSGU), which are general
structures for learning long term dependencies. Compared to traditional Long
Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), both structures
require fewer parameters and less computation time in sequence classification
tasks. Unlike GRU and LSTM, which require more than one gates to control
information flow in the network, SGU and DSGU only use one multiplicative gate
to control the flow of information. We show that this difference can accelerate
the learning speed in tasks that require long dependency information. We also
show that DSGU is more numerically stable than SGU. In addition, we also
propose a standard way of representing inner structure of RNN called RNN
Conventional Graph (RCG), which helps analyzing the relationship between input
units and hidden units of RNN.
