  Data noising is an effective technique for regularizing neural network
models. While noising is widely adopted in application domains such as vision
and speech, commonly used noising primitives have not been developed for
discrete sequence-level settings such as language modeling. In this paper, we
derive a connection between input noising in neural network language models and
smoothing in $n$-gram models. Using this connection, we draw upon ideas from
smoothing to develop effective noising schemes. We demonstrate performance
gains when applying the proposed schemes to language modeling and machine
translation. Finally, we provide empirical analysis validating the relationship
between noising and smoothing.
