  In this brief paper, we investigate online training of Long Short Term Memory
(LSTM) architectures in a distributed network of nodes, where each node employs
an LSTM based structure for online regression. In particular, each node
sequentially receives a variable length data sequence with its label and can
only exchange information with its neighbors to train the LSTM architecture. We
first provide a generic LSTM based regression structure for each node. In order
to train this structure, we put the LSTM equations in a nonlinear state space
form for each node and then introduce a highly effective and efficient
Distributed Particle Filtering (DPF) based training algorithm. We also
introduce a Distributed Extended Kalman Filtering (DEKF) based training
algorithm for comparison. Here, our DPF based training algorithm guarantees
convergence to the performance of the optimal LSTM coefficients in the mean
square error (MSE) sense under certain conditions. We achieve this performance
with communication and computational complexity in the order of the first order
gradient based methods. Through both simulated and real life examples, we
illustrate significant performance improvements with respect to the state of
the art methods.
