  There is a large variety of objects and appliances in human environments,
such as stoves, coffee dispensers, juice extractors, and so on. It is
challenging for a roboticist to program a robot for each of these object types
and for each of their instantiations. In this work, we present a novel approach
to manipulation planning based on the idea that many household objects share
similarly-operated object parts. We formulate the manipulation planning as a
structured prediction problem and design a deep learning model that can handle
large noise in the manipulation demonstrations and learns features from three
different modalities: point-clouds, language and trajectory. In order to
collect a large number of manipulation demonstrations for different objects, we
developed a new crowd-sourcing platform called Robobarista. We test our model
on our dataset consisting of 116 objects with 249 parts along with 250 language
instructions, for which there are 1225 crowd-sourced manipulation
demonstrations. We further show that our robot can even manipulate objects it
has never seen before.
