  We tackle sequential learning under label noise in applications where a human
supervisor can be queried to relabel suspicious examples. Existing approaches
are flawed, in that they only relabel incoming examples that look "suspicious"
to the model. As a consequence, those mislabeled examples that elude (or don't
undergo) this cleaning step end up tainting the training data and the model
with no further chance of being cleaned. We propose Cincer, a novel approach
that cleans both new and past data by identifying pairs of mutually
incompatible examples. Whenever it detects a suspicious example, Cincer
identifies a counter-example in the training set that -- according to the model
-- is maximally incompatible with the suspicious example, and asks the
annotator to relabel either or both examples, resolving this possible
inconsistency. The counter-examples are chosen to be maximally incompatible, so
to serve as explanations of the model's suspicion, and highly influential, so
to convey as much information as possible if relabeled. Cincer achieves this by
leveraging an efficient and robust approximation of influence functions based
on the Fisher information matrix (FIM). Our extensive empirical evaluation
shows that clarifying the reasons behind the model's suspicions by cleaning the
counter-examples helps in acquiring substantially better data and models,
especially when paired with our FIM approximation.
