  Invariance describes transformations that do not alter data's underlying
semantics. Neural networks that preserve natural invariance capture good
inductive biases and achieve superior performance. Hence, modern networks are
handcrafted to handle well-known invariances (ex. translations). We propose a
framework to learn novel network architectures that capture data-dependent
invariances via pruning. Our learned architectures consistently outperform
dense neural networks on both vision and tabular datasets in both efficiency
and effectiveness. We demonstrate our framework on multiple deep learning
models across 3 vision and 40 tabular datasets.
