  Distantly-labeled data can be used to scale up training of statistical
models, but it is typically noisy and that noise can vary with the distant
labeling technique. In this work, we propose a two-stage procedure for handling
this type of data: denoise it with a learned model, then train our final model
on clean and denoised distant data with standard supervised training. Our
denoising approach consists of two parts. First, a filtering function discards
examples from the distantly labeled data that are wholly unusable. Second, a
relabeling function repairs noisy labels for the retained examples. Each of
these components is a model trained on synthetically-noised examples generated
from a small manually-labeled set. We investigate this approach on the
ultra-fine entity typing task of Choi et al. (2018). Our baseline model is an
extension of their model with pre-trained ELMo representations, which already
achieves state-of-the-art performance. Adding distant data that has been
denoised with our learned models gives further performance gains over this base
model, outperforming models trained on raw distant data or
heuristically-denoised distant data.
