  Recurrent neural networks (RNN) have been successfully applied to various
sequential decision-making tasks, natural language processing applications, and
time-series predictions. Such networks are usually trained through
back-propagation through time (BPTT) which is prohibitively expensive,
especially when the length of the time dependencies and the number of hidden
neurons increase. To reduce the training time, extreme learning machines (ELMs)
have been recently applied to RNN training, reaching a 99\% speedup on some
applications. Due to its non-iterative nature, ELM training, when parallelized,
has the potential to reach higher speedups than BPTT.
  In this work, we present \opt, an optimized parallel RNN training algorithm
based on ELM that takes advantage of the GPU shared memory and of parallel QR
factorization algorithms to efficiently reach optimal solutions. The
theoretical analysis of the proposed algorithm is presented on six RNN
architectures, including LSTM and GRU, and its performance is empirically
tested on ten time-series prediction applications. \opt~is shown to reach up to
845 times speedup over its sequential counterpart and to require up to 20x less
time to train than parallel BPTT.
