  We present a novel LSTM cell architecture capable of learning both intra- and
inter-perspective relationships available in visual sequences captured from
multiple perspectives. Our architecture adopts a novel recurrent joint learning
strategy that uses additional gates and memories at the cell level. We
demonstrate that by using the proposed cell to create a network, more effective
and richer visual representations are learned for recognition tasks. We
validate the performance of our proposed architecture in the context of two
multi-perspective visual recognition tasks namely lip reading and face
recognition. Three relevant datasets are considered and the results are
compared against fusion strategies, other existing multi-input LSTM
architectures, and alternative recognition solutions. The experiments show the
superior performance of our solution over the considered benchmarks, both in
terms of recognition accuracy and complexity. We make our code publicly
available at https://github.com/arsm/MPLSTM.
