  For effective human-robot interaction, it is important that a robotic
assistant can forecast the next action a human will consider in a given task.
Unfortunately, real-world tasks are often very long, complex, and repetitive;
as a result forecasting is not trivial. In this paper, we propose a novel deep
recurrent architecture that takes as input features from a two-stream Residual
action recognition framework, and learns to estimate the progress of human
activities from video sequences -- this surrogate progress estimation task
implicitly learns a temporal task grammar with respect to which activities can
be localized and forecasted. To learn the task grammar, we propose a stacked
LSTM based multi-granularity progress estimation framework that uses a novel
cumulative Euclidean loss as objective. To demonstrate the effectiveness of our
proposed architecture, we showcase experiments on two challenging robotic
assistive tasks, namely (i) assembling an Ikea table from its constituents, and
(ii) changing the tires of a car. Our results demonstrate that learning task
grammars offers highly discriminative cues improving the forecasting accuracy
by more than 9% over the baseline two-stream forecasting model, while also
outperforming other competitive schemes.
