  Complementary to finding good general word embeddings, an important question
for representation learning is to find dynamic word embeddings, e.g., across
time or domain. Current methods do not offer a way to use or predict
information on structure between sub-corpora, time or domain and dynamic
embeddings can only be compared after post-alignment. We propose novel word
embedding methods that provide general word representations for the whole
corpus, domain-specific representations for each sub-corpus, sub-corpus
structure, and embedding alignment simultaneously. We present an empirical
evaluation on New York Times articles and two English Wikipedia datasets with
articles on science and philosophy. Our method, called Word2Vec with Structure
Prediction (W2VPred), provides better performance than baselines in terms of
the general analogy tests, domain-specific analogy tests, and multiple specific
word embedding evaluations as well as structure prediction performance when no
structure is given a priori. As a use case in the field of Digital Humanities
we demonstrate how to raise novel research questions for high literature from
the German Text Archive.
