  Distant supervision (DS) is a strong way to expand the datasets for enhancing
relation extraction (RE) models but often suffers from high label noise.
Current works based on attention, reinforcement learning, or GAN are black-box
models so they neither provide meaningful interpretation of sample selection in
DS nor stability on different domains. On the contrary, this work proposes a
novel model-agnostic instance sampling method for DS by influence function
(IF), namely REIF. Our method identifies favorable/unfavorable instances in the
bag based on IF, then does dynamic instance sampling. We design a fast
influence sampling algorithm that reduces the computational complexity from
$\mathcal{O}(mn)$ to $\mathcal{O}(1)$, with analyzing its robustness on the
selected sampling function. Experiments show that by simply sampling the
favorable instances during training, REIF is able to win over a series of
baselines that have complicated architectures. We also demonstrate that REIF
can support interpretable instance selection.
