  We develop a learning framework for building deformable templates, which play
a fundamental role in many image analysis and computational anatomy tasks.
Conventional methods for template creation and image alignment to the template
have undergone decades of rich technical development. In these frameworks,
templates are constructed using an iterative process of template estimation and
alignment, which is often computationally very expensive. Due in part to this
shortcoming, most methods compute a single template for the entire population
of images, or a few templates for specific sub-groups of the data. In this
work, we present a probabilistic model and efficient learning strategy that
yields either universal or conditional templates, jointly with a neural network
that provides efficient alignment of the images to these templates. We
demonstrate the usefulness of this method on a variety of domains, with a
special focus on neuroimaging. This is particularly useful for clinical
applications where a pre-existing template does not exist, or creating a new
one with traditional methods can be prohibitively expensive. Our code and
atlases are available online as part of the VoxelMorph library at
http://voxelmorph.csail.mit.edu.
