  Learning from human demonstrations is an emerging trend for designing
intelligent robotic systems. However, previous methods typically regard videos
as instructions, simply dividing them into action sequences for robotic
repetition, which poses obstacles to generalization to diverse tasks or object
instances. In this paper, we propose a different perspective, considering human
demonstration videos not as mere instructions, but as a source of knowledge for
robots. Motivated by this perspective and the remarkable comprehension and
generalization capabilities exhibited by large language models (LLMs), we
propose DigKnow, a method that DIstills Generalizable KNOWledge with a
hierarchical structure. Specifically, DigKnow begins by converting human
demonstration video frames into observation knowledge. This knowledge is then
subjected to analysis to extract human action knowledge and further distilled
into pattern knowledge compassing task and object instances, resulting in the
acquisition of generalizable knowledge with a hierarchical structure. In
settings with different tasks or object instances, DigKnow retrieves relevant
knowledge for the current task and object instances. Subsequently, the
LLM-based planner conducts planning based on the retrieved knowledge, and the
policy executes actions in line with the plan to achieve the designated task.
Utilizing the retrieved knowledge, we validate and rectify planning and
execution outcomes, resulting in a substantial enhancement of the success rate.
Experimental results across a range of tasks and scenes demonstrate the
effectiveness of this approach in facilitating real-world robots to accomplish
tasks with the knowledge derived from human demonstrations.
