  This paper presents a novel method to predict future human activities from
partially observed RGB-D videos. Human activity prediction is generally
difficult due to its non-Markovian property and the rich context between human
and environments.
  We use a stochastic grammar model to capture the compositional structure of
events, integrating human actions, objects, and their affordances. We represent
the event by a spatial-temporal And-Or graph (ST-AOG). The ST-AOG is composed
of a temporal stochastic grammar defined on sub-activities, and spatial graphs
representing sub-activities that consist of human actions, objects, and their
affordances. Future sub-activities are predicted using the temporal grammar and
Earley parsing algorithm. The corresponding action, object, and affordance
labels are then inferred accordingly. Extensive experiments are conducted to
show the effectiveness of our model on both semantic event parsing and future
activity prediction.
