  Distant supervision significantly reduces human efforts in building training
data for many classification tasks. While promising, this technique often
introduces noise to the generated training data, which can severely affect the
model performance. In this paper, we take a deep look at the application of
distant supervision in relation extraction. We show that the dynamic transition
matrix can effectively characterize the noise in the training data built by
distant supervision. The transition matrix can be effectively trained using a
novel curriculum learning based method without any direct supervision about the
noise. We thoroughly evaluate our approach under a wide range of extraction
scenarios. Experimental results show that our approach consistently improves
the extraction results and outperforms the state-of-the-art in various
evaluation scenarios.
