  Manipulation planning is the problem of finding a sequence of robot
configurations that involves interactions with objects in the scene, e.g.,
grasping and placing an object, or more general tool-use. To achieve such
interactions, traditional approaches require hand-engineering of object
representations and interaction constraints, which easily becomes tedious when
complex objects/interactions are considered. Inspired by recent advances in 3D
modeling, e.g. NeRF, we propose a method to represent objects as continuous
functions upon which constraint features are defined and jointly trained. In
particular, the proposed pixel-aligned representation is directly inferred from
images with known camera geometry and naturally acts as a perception component
in the whole manipulation pipeline, thereby enabling long-horizon planning only
from visual input. Project page:
https://sites.google.com/view/deep-visual-constraints
