  Recent work exhibited that distributed word representations are good at
capturing linguistic regularities in language. This allows vector-oriented
reasoning based on simple linear algebra between words. Since many different
methods have been proposed for learning document representations, it is natural
to ask whether there is also linear structure in these learned representations
to allow similar reasoning at document level. To answer this question, we
design a new document analogy task for testing the semantic regularities in
document representations, and conduct empirical evaluations over several
state-of-the-art document representation models. The results reveal that neural
embedding based document representations work better on this analogy task than
conventional methods, and we provide some preliminary explanations over these
observations.
