  We propose a novel neural memory network based framework for future action
sequence forecasting. This is a challenging task where we have to consider
short-term, within sequence relationships as well as relationships in between
sequences, to understand how sequences of actions evolve over time. To capture
these relationships effectively, we introduce neural memory networks to our
modelling scheme. We show the significance of using two input streams, the
observed frames and the corresponding action labels, which provide different
information cues for our prediction task. Furthermore, through the proposed
method we effectively map the long-term relationships among individual input
sequences through separate memory modules, which enables better fusion of the
salient features. Our method outperforms the state-of-the-art approaches by a
large margin on two publicly available datasets: Breakfast and 50 Salads.
