  Much research in machine learning involves finding appropriate inductive
biases (e.g. convolutional neural networks, momentum-based optimizers,
transformers) to promote generalization on tasks. However, quantification of
the amount of inductive bias associated with these architectures and
hyperparameters has been limited. We propose a novel method for efficiently
computing the inductive bias required for generalization on a task with a fixed
training data budget; formally, this corresponds to the amount of information
required to specify well-generalizing models within a specific hypothesis space
of models. Our approach involves modeling the loss distribution of random
hypotheses drawn from a hypothesis space to estimate the required inductive
bias for a task relative to these hypotheses. Unlike prior work, our method
provides a direct estimate of inductive bias without using bounds and is
applicable to diverse hypothesis spaces. Moreover, we derive approximation
error bounds for our estimation approach in terms of the number of sampled
hypotheses. Consistent with prior results, our empirical results demonstrate
that higher dimensional tasks require greater inductive bias. We show that
relative to other expressive model classes, neural networks as a model class
encode large amounts of inductive bias. Furthermore, our measure quantifies the
relative difference in inductive bias between different neural network
architectures. Our proposed inductive bias metric provides an
information-theoretic interpretation of the benefits of specific model
architectures for certain tasks and provides a quantitative guide to developing
tasks requiring greater inductive bias, thereby encouraging the development of
more powerful inductive biases.
