  Training set bugs are flaws in the data that adversely affect machine
learning. The training set is usually too large for man- ual inspection, but
one may have the resources to verify a few trusted items. The set of trusted
items may not by itself be adequate for learning, so we propose an algorithm
that uses these items to identify bugs in the training set and thus im- proves
learning. Specifically, our approach seeks the smallest set of changes to the
training set labels such that the model learned from this corrected training
set predicts labels of the trusted items correctly. We flag the items whose
labels are changed as potential bugs, whose labels can be checked for veracity
by human experts. To find the bugs in this way is a challenging combinatorial
bilevel optimization problem, but it can be relaxed into a continuous
optimization problem. Ex- periments on toy and real data demonstrate that our
approach can identify training set bugs effectively and suggest appro- priate
changes to the labels. Our algorithm is a step toward trustworthy machine
learning.
