  Programs offer compactness and structure that makes them an attractive
representation for visual data. We explore how code rewriting can be used to
improve systems for inferring programs from visual data. We first propose
Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised
bootstrapped learning. SIRI sparsely applies code rewrite operations over a
dataset of training programs, injecting the improved programs back into the
training set. We design a family of rewriters for visual programming domains:
parameter optimization, code pruning, and code grafting. For three shape
programming languages in 2D and 3D, we show that using SIRI with our family of
rewriters improves performance: better reconstructions and faster convergence
rates, compared with bootstrapped learning methods that do not use rewriters or
use them naively. Finally, we demonstrate that our family of rewriters can be
effectively used at test time to improve the output of SIRI predictions. For 2D
and 3D CSG, we outperform or match the reconstruction performance of recent
domain-specific neural architectures, while producing more parsimonious
programs that use significantly fewer primitives.
