  As the first step in automated natural language processing, representing
words and sentences is of central importance and has attracted significant
research attention. Different approaches, from the early one-hot and
bag-of-words representation to more recent distributional dense and sparse
representations, were proposed. Despite the successful results that have been
achieved, such vectors tend to consist of uninterpretable components and face
nontrivial challenge in both memory and computational requirement in practical
applications. In this paper, we designed a novel representation model that
projects dense word vectors into a higher dimensional space and favors a highly
sparse and binary representation of word vectors with potentially interpretable
components, while trying to maintain pairwise inner products between original
vectors as much as possible. Computationally, our model is relaxed as a
symmetric non-negative matrix factorization problem which admits a fast yet
effective solution. In a series of empirical evaluations, the proposed model
exhibited consistent improvement and high potential in practical applications.
