  Character-based Neural Network Language Models (NNLM) have the advantage of
smaller vocabulary and thus faster training times in comparison to NNLMs based
on multi-character units. However, in low-resource scenarios, both the
character and multi-character NNLMs suffer from data sparsity. In such
scenarios, cross-lingual transfer has improved multi-character NNLM performance
by allowing information transfer from a source to the target language. In the
same vein, we propose to use cross-lingual transfer for character NNLMs applied
to low-resource Automatic Speech Recognition (ASR). However, applying
cross-lingual transfer to character NNLMs is not as straightforward. We observe
that relatedness of the source language plays an important role in
cross-lingual pretraining of character NNLMs. We evaluate this aspect on ASR
tasks for two target languages: Finnish (with English and Estonian as source)
and Swedish (with Danish, Norwegian, and English as source). Prior work has
observed no difference between using the related or unrelated language for
multi-character NNLMs. We, however, show that for character-based NNLMs, only
pretraining with a related language improves the ASR performance, and using an
unrelated language may deteriorate it. We also observe that the benefits are
larger when there is much lesser target data than source data.
