  We propose a novel non-parametric/un-trainable language model, named
Non-Parametric Pairwise Attention Random Walk Model (NoPPA), to generate
sentence embedding only with pre-trained word embedding and pre-counted word
frequency. To the best we know, this study is the first successful attempt to
break the constraint on bag-of-words assumption with a non-parametric attention
mechanism. We evaluate our method on eight different downstream classification
tasks. The experiment results show that NoPPA outperforms all kinds of
bag-of-words-based methods in each dataset and provides a comparable or better
performance than the state-of-the-art non-parametric methods on average.
Furthermore, visualization supports that NoPPA can understand contextual
topics, common phrases, and word causalities. Our model is available at
https://github.com/JacksonWuxs/NoPPA.
