  Traditionally, data valuation (DV) is posed as a problem of equitably
splitting the validation performance of a learning algorithm among the training
data. As a result, the calculated data values depend on many design choices of
the underlying learning algorithm. However, this dependence is undesirable for
many DV use cases, such as setting priorities over different data sources in a
data acquisition process and informing pricing mechanisms in a data
marketplace. In these scenarios, data needs to be valued before the actual
analysis and the choice of the learning algorithm is still undetermined then.
Another side-effect of the dependence is that to assess the value of individual
points, one needs to re-run the learning algorithm with and without a point,
which incurs a large computation burden. This work leapfrogs over the current
limits of data valuation methods by introducing a new framework that can value
training data in a way that is oblivious to the downstream learning algorithm.
Our main results are as follows. (1) We develop a proxy for the validation
performance associated with a training set based on a non-conventional
class-wise Wasserstein distance between training and validation sets. We show
that the distance characterizes the upper bound of the validation performance
for any given model under certain Lipschitz conditions. (2) We develop a novel
method to value individual data based on the sensitivity analysis of the
class-wise Wasserstein distance. Importantly, these values can be directly
obtained for free from the output of off-the-shelf optimization solvers when
computing the distance. (3) We evaluate our new data valuation framework over
various use cases related to detecting low-quality data and show that,
surprisingly, the learning-agnostic feature of our framework enables a
significant improvement over SOTA performance while being orders of magnitude
faster.
