  Robotic manipulation can be formulated as inducing a sequence of spatial
displacements: where the space being moved can encompass an object, part of an
object, or end effector. In this work, we propose the Transporter Network, a
simple model architecture that rearranges deep features to infer spatial
displacements from visual input - which can parameterize robot actions. It
makes no assumptions of objectness (e.g. canonical poses, models, or
keypoints), it exploits spatial symmetries, and is orders of magnitude more
sample efficient than our benchmarked alternatives in learning vision-based
manipulation tasks: from stacking a pyramid of blocks, to assembling kits with
unseen objects; from manipulating deformable ropes, to pushing piles of small
objects with closed-loop feedback. Our method can represent complex multi-modal
policy distributions and generalizes to multi-step sequential tasks, as well as
6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns
faster and generalizes better than a variety of end-to-end baselines, including
policies that use ground-truth object poses. We validate our methods with
hardware in the real world. Experiment videos and code are available at
https://transporternets.github.io
