  We consider the problem of learning distributed representations for documents
in data streams. The documents are represented as low-dimensional vectors and
are jointly learned with distributed vector representations of word tokens
using a hierarchical framework with two embedded neural language models. In
particular, we exploit the context of documents in streams and use one of the
language models to model the document sequences, and the other to model word
sequences within them. The models learn continuous vector representations for
both word tokens and documents such that semantically similar documents and
words are close in a common vector space. We discuss extensions to our model,
which can be applied to personalized recommendation and social relationship
mining by adding further user layers to the hierarchy, thus learning
user-specific vectors to represent individual preferences. We validated the
learned representations on a public movie rating data set from MovieLens, as
well as on a large-scale Yahoo News data comprising three months of user
activity logs collected on Yahoo servers. The results indicate that the
proposed model can learn useful representations of both documents and word
tokens, outperforming the current state-of-the-art by a large margin.
