  The recently proposed Unbiased Online Recurrent Optimization algorithm (UORO,
arXiv:1702.05043) uses an unbiased approximation of RTRL to achieve fully
online gradient-based learning in RNNs. In this work we analyze the variance of
the gradient estimate computed by UORO, and propose several possible changes to
the method which reduce this variance both in theory and practice. We also
contribute significantly to the theoretical and intuitive understanding of UORO
(and its existing variance reduction technique), and demonstrate a fundamental
connection between its gradient estimate and the one that would be computed by
REINFORCE if small amounts of noise were added to the RNN's hidden units.
