  Manipulation and assembly tasks require non-trivial planning of actions
depending on the environment and the final goal. Previous work in this domain
often assembles particular instances of objects from known sets of primitives.
In contrast, we aim to handle varying sets of primitives and to construct
different objects of a shape category. Given a single object instance of a
category, e.g. an arch, and a binary shape classifier, we learn a visual policy
to assemble other instances of the same category. In particular, we propose a
disassembly procedure and learn a state policy that discovers new object
instances and their assembly plans in state space. We then render simulated
states in the observation space and learn a heatmap representation to predict
alternative actions from a given input image. To validate our approach, we
first demonstrate its efficiency for building object categories in state space.
We then show the success of our visual policies for building arches from
different primitives. Moreover, we demonstrate (i) the reactive ability of our
method to re-assemble objects using additional primitives and (ii) the robust
performance of our policy for unseen primitives resembling building blocks used
during training. Our visual assembly policies are trained with no real images
and reach up to 95% success rate when evaluated on a real robot.
