  The goal of language modeling techniques is to capture the statistical and
structural properties of natural languages from training corpora. This task
typically involves the learning of short range dependencies, which generally
model the syntactic properties of a language and/or long range dependencies,
which are semantic in nature. We propose in this paper a new multi-span
architecture, which separately models the short and long context information
while it dynamically merges them to perform the language modeling task. This is
done through a novel recurrent Long-Short Range Context (LSRC) network, which
explicitly models the local (short) and global (long) context using two
separate hidden states that evolve in time. This new architecture is an
adaptation of the Long-Short Term Memory network (LSTM) to take into account
the linguistic properties. Extensive experiments conducted on the Penn Treebank
(PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a
significant reduction of the perplexity when compared to state-of-the-art
language modeling techniques.
