  We present a strategy for designing and building very general robot
manipulation systems involving the integration of a general-purpose
task-and-motion planner with engineered and learned perception modules that
estimate properties and affordances of unknown objects. Such systems are
closed-loop policies that map from RGB images, depth images, and robot joint
encoder measurements to robot joint position commands. We show that following
this strategy a task-and-motion planner can be used to plan intelligent
behaviors even in the absence of a priori knowledge regarding the set of
manipulable objects, their geometries, and their affordances. We explore
several different ways of implementing such perceptual modules for
segmentation, property detection, shape estimation, and grasp generation. We
show how these modules are integrated within the PDDLStream task and motion
planning framework. Finally, we demonstrate that this strategy can enable a
single system to perform a wide variety of real-world multi-step manipulation
tasks, generalizing over a broad class of objects, object arrangements, and
goals, without any prior knowledge of the environment and without re-training.
