  Learning new skills by observing humans' behaviors is an essential capability
of AI. In this work, we leverage instructional videos to study humans'
decision-making processes, focusing on learning a model to plan goal-directed
actions in real-life videos. In contrast to conventional action recognition,
goal-directed actions are based on expectations of their outcomes requiring
causal knowledge of potential consequences of actions. Thus, integrating the
environment structure with goals is critical for solving this task. Previous
works learn a single world model will fail to distinguish various tasks,
resulting in an ambiguous latent space; planning through it will gradually
neglect the desired outcomes since the global information of the future goal
degrades quickly as the procedure evolves. We address these limitations with a
new formulation of procedure planning and propose novel algorithms to model
human behaviors through Bayesian Inference and model-based Imitation Learning.
Experiments conducted on real-world instructional videos show that our method
can achieve state-of-the-art performance in reaching the indicated goals.
Furthermore, the learned contextual information presents interesting features
for planning in a latent space.
