  Gradient control plays an important role in feed-forward networks applied to
various computer vision tasks. Previous work has shown that Recurrent Highway
Networks minimize the problem of vanishing or exploding gradients. They achieve
this by setting the eigenvalues of the temporal Jacobian to 1 across the time
steps. In this work, batch normalized recurrent highway networks are proposed
to control the gradient flow in an improved way for network convergence.
Specifically, the introduced model can be formed by batch normalizing the
inputs at each recurrence loop. The proposed model is tested on an image
captioning task using MSCOCO dataset. Experimental results indicate that the
batch normalized recurrent highway networks converge faster and performs better
compared with the traditional LSTM and RHN based models.
