  We leverage the Neural Tangent Kernel and its equivalence to training
infinitely-wide neural networks to devise $\infty$-AE: an autoencoder with
infinitely-wide bottleneck layers. The outcome is a highly expressive yet
simplistic recommendation model with a single hyper-parameter and a closed-form
solution. Leveraging $\infty$-AE's simplicity, we also develop Distill-CF for
synthesizing tiny, high-fidelity data summaries which distill the most
important knowledge from the extremely large and sparse user-item interaction
matrix for efficient and accurate subsequent data-usage like model training,
inference, architecture search, etc. This takes a data-centric approach to
recommendation, where we aim to improve the quality of logged user-feedback
data for subsequent modeling, independent of the learning algorithm. We
particularly utilize the concept of differentiable Gumbel-sampling to handle
the inherent data heterogeneity, sparsity, and semi-structuredness, while being
scalable to datasets with hundreds of millions of user-item interactions. Both
of our proposed approaches significantly outperform their respective
state-of-the-art and when used together, we observe 96-105% of $\infty$-AE's
performance on the full dataset with as little as 0.1% of the original dataset
size, leading us to explore the counter-intuitive question: Is more data what
you need for better recommendation?
