  An important goal of AutoML is to automate-away the design of neural networks
on new tasks in under-explored domains. Motivated by this goal, we study the
problem of enabling users to discover the right neural operations given data
from their specific domain. We introduce a search space of operations called
XD-Operations that mimic the inductive bias of standard multi-channel
convolutions while being much more expressive: we prove that it includes many
named operations across multiple application areas. Starting with any standard
backbone such as ResNet, we show how to transform it into a search space over
XD-operations and how to traverse the space using a simple weight-sharing
scheme. On a diverse set of tasks -- solving PDEs, distance prediction for
protein folding, and music modeling -- our approach consistently yields models
with lower error than baseline networks and often even lower error than
expert-designed domain-specific approaches.
