  This paper aims at task-oriented action prediction, i.e., predicting a
sequence of actions towards accomplishing a specific task under a certain
scene, which is a new problem in computer vision research. The main challenges
lie in how to model task-specific knowledge and integrate it in the learning
procedure. In this work, we propose to train a recurrent long-short term memory
(LSTM) network for handling this problem, i.e., taking a scene image (including
pre-located objects) and the specified task as input and recurrently predicting
action sequences. However, training such a network usually requires large
amounts of annotated samples for covering the semantic space (e.g., diverse
action decomposition and ordering). To alleviate this issue, we introduce a
temporal And-Or graph (AOG) for task description, which hierarchically
represents a task into atomic actions. With this AOG representation, we can
produce many valid samples (i.e., action sequences according with common sense)
by training another auxiliary LSTM network with a small set of annotated
samples. And these generated samples (i.e., task-oriented action sequences)
effectively facilitate training the model for task-oriented action prediction.
In the experiments, we create a new dataset containing diverse daily tasks and
extensively evaluate the effectiveness of our approach.
