  Word embedding is a key component in many downstream applications in
processing natural languages. Existing approaches often assume the existence of
a large collection of text for learning effective word embedding. However, such
a corpus may not be available for some low-resource languages. In this paper,
we study how to effectively learn a word embedding model on a corpus with only
a few million tokens. In such a situation, the co-occurrence matrix is sparse
as the co-occurrences of many word pairs are unobserved. In contrast to
existing approaches often only sample a few unobserved word pairs as negative
samples, we argue that the zero entries in the co-occurrence matrix also
provide valuable information. We then design a Positive-Unlabeled Learning
(PU-Learning) approach to factorize the co-occurrence matrix and validate the
proposed approaches in four different languages.
