  Training data attribution (TDA) methods aim to quantify the influence of
individual training data points on the model predictions, with broad
applications in data-centric AI, such as mislabel detection, data selection,
and copyright compensation. However, existing methods in this field, which can
be categorized as retraining-based and gradient-based, have struggled with the
trade-off between computational efficiency and attribution efficacy.
Retraining-based methods can accurately attribute complex non-convex models but
are computationally prohibitive, while gradient-based methods are efficient but
often fail for non-convex models. Recent research has shown that augmenting
gradient-based methods with ensembles of multiple independently trained models
can achieve significantly better attribution efficacy. However, this approach
remains impractical for very large-scale applications.
  In this work, we discover that expensive, fully independent training is
unnecessary for ensembling the gradient-based methods, and we propose two
efficient ensemble strategies, DROPOUT ENSEMBLE and LORA ENSEMBLE, alternative
to naive independent ensemble. These strategies significantly reduce training
time (up to 80%), serving time (up to 60%), and space cost (up to 80%) while
maintaining similar attribution efficacy to the naive independent ensemble. Our
extensive experimental results demonstrate that the proposed strategies are
effective across multiple TDA methods on diverse datasets and models, including
generative settings, significantly advancing the Pareto frontier of TDA methods
with better computational efficiency and attribution efficacy.
