  We present a system to infer and execute a human-readable program from a
real-world demonstration. The system consists of a series of neural networks to
perform perception, program generation, and program execution. Leveraging
convolutional pose machines, the perception network reliably detects the
bounding cuboids of objects in real images even when severely occluded, after
training only on synthetic images using domain randomization. To increase the
applicability of the perception network to new scenarios, the network is
formulated to predict in image space rather than in world space. Additional
networks detect relationships between objects, generate plans, and determine
actions to reproduce a real-world demonstration. The networks are trained
entirely in simulation, and the system is tested in the real world on the
pick-and-place problem of stacking colored cubes using a Baxter robot.
