  Humans are capable of building holistic representations for images at various
levels, from local objects, to pairwise relations, to global structures. The
interpretation of structures involves reasoning over repetition and symmetry of
the objects in the image. In this paper, we present the Program-Guided Image
Manipulator (PG-IM), inducing neuro-symbolic program-like representations to
represent and manipulate images. Given an image, PG-IM detects repeated
patterns, induces symbolic programs, and manipulates the image using a neural
network that is guided by the program. PG-IM learns from a single image,
exploiting its internal statistics. Despite trained only on image inpainting,
PG-IM is directly capable of extrapolation and regularity editing in a unified
framework. Extensive experiments show that PG-IM achieves superior performance
on all the tasks.
