  No free lunch theorems for supervised learning state that no learner can
solve all problems or that all learners achieve exactly the same accuracy on
average over a uniform distribution on learning problems. Accordingly, these
theorems are often referenced in support of the notion that individual problems
require specially tailored inductive biases. While virtually all uniformly
sampled datasets have high complexity, real-world problems disproportionately
generate low-complexity data, and we argue that neural network models share
this same preference, formalized using Kolmogorov complexity. Notably, we show
that architectures designed for a particular domain, such as computer vision,
can compress datasets on a variety of seemingly unrelated domains. Our
experiments show that pre-trained and even randomly initialized language models
prefer to generate low-complexity sequences. Whereas no free lunch theorems
seemingly indicate that individual problems require specialized learners, we
explain how tasks that often require human intervention such as picking an
appropriately sized model when labeled data is scarce or plentiful can be
automated into a single learning algorithm. These observations justify the
trend in deep learning of unifying seemingly disparate problems with an
increasingly small set of machine learning models.
