  Anticipating the future actions of a human is a widely studied problem in
robotics that requires spatio-temporal reasoning. In this work we propose a
deep learning approach for anticipation in sensory-rich robotics applications.
We introduce a sensory-fusion architecture which jointly learns to anticipate
and fuse information from multiple sensory streams. Our architecture consists
of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM)
units to capture long temporal dependencies. We train our architecture in a
sequence-to-sequence prediction manner, and it explicitly learns to predict the
future given only a partial temporal context. We further introduce a novel loss
layer for anticipation which prevents over-fitting and encourages early
anticipation. We use our architecture to anticipate driving maneuvers several
seconds before they happen on a natural driving data set of 1180 miles. The
context for maneuver anticipation comes from multiple sensors installed on the
vehicle. Our approach shows significant improvement over the state-of-the-art
in maneuver anticipation by increasing the precision from 77.4% to 90.5% and
recall from 71.2% to 87.4%.
