  We develop a framework for combining differentiable programming languages
with neural networks. Using this framework we create end-to-end trainable
systems that learn to write interpretable algorithms with perceptual
components. We explore the benefits of inductive biases for strong
generalization and modularity that come from the program-like structure of our
models. In particular, modularity allows us to learn a library of (neural)
functions which grows and improves as more tasks are solved. Empirically, we
show that this leads to lifelong learning systems that transfer knowledge to
new tasks more effectively than baselines.
