  Efficient representation of text documents is an important building block in
many NLP tasks. Research on long text categorization has shown that simple
weighted averaging of word vectors for sentence representation often
outperforms more sophisticated neural models. Recently proposed Sparse
Composite Document Vector (SCDV) (Mekala et. al, 2017) extends this approach
from sentences to documents using soft clustering over word vectors. However,
SCDV disregards the multi-sense nature of words, and it also suffers from the
curse of higher dimensionality. In this work, we address these shortcomings and
propose SCDV-MS. SCDV-MS utilizes multi-sense word embeddings and learns a
lower dimensional manifold. Through extensive experiments on multiple
real-world datasets, we show that SCDV-MS embeddings outperform previous
state-of-the-art embeddings on multi-class and multi-label text categorization
tasks. Furthermore, SCDV-MS embeddings are more efficient than SCDV in terms of
time and space complexity on textual classification tasks.
