  Point clouds are a widely available and canonical data modality which convey
the 3D geometry of a scene. Despite significant progress in classification and
segmentation from point clouds, policy learning from such a modality remains
challenging, and most prior works in imitation learning focus on learning
policies from images or state information. In this paper, we propose a novel
framework for learning policies from point clouds for robotic manipulation with
tools. We use a novel neural network, ToolFlowNet, which predicts dense
per-point flow on the tool that the robot controls, and then uses the flow to
derive the transformation that the robot should execute. We apply this
framework to imitation learning of challenging deformable object manipulation
tasks with continuous movement of tools, including scooping and pouring, and
demonstrate significantly improved performance over baselines which do not use
flow. We perform 50 physical scooping experiments with ToolFlowNet and attain
82% scooping success. See https://tinyurl.com/toolflownet for supplementary
material.
