  In this work, we study the problem of how to leverage instructional videos to
facilitate the understanding of human decision-making processes, focusing on
training a model with the ability to plan a goal-directed procedure from
real-world videos. Learning structured and plannable state and action spaces
directly from unstructured videos is the key technical challenge of our task.
There are two problems: first, the appearance gap between the training and
validation datasets could be large for unstructured videos; second, these gaps
lead to decision errors that compound over the steps. We address these
limitations with Planning Transformer (PlaTe), which has the advantage of
circumventing the compounding prediction errors that occur with single-step
models during long model-based rollouts. Our method simultaneously learns the
latent state and action information of assigned tasks and the representations
of the decision-making process from human demonstrations. Experiments conducted
on real-world instructional videos and an interactive environment show that our
method can achieve a better performance in reaching the indicated goal than
previous algorithms. We also validated the possibility of applying procedural
tasks on a UR-5 platform. We make our code publicly available and support
academic research purposes.
