  Word embeddings resulting from neural language models have been shown to be
successful for a large variety of NLP tasks. However, such architecture might
be difficult to train and time-consuming. Instead, we propose to drastically
simplify the word embeddings computation through a Hellinger PCA of the word
co-occurence matrix. We compare those new word embeddings with some well-known
embeddings on NER and movie review tasks and show that we can reach similar or
even better performance. Although deep learning is not really necessary for
generating good word embeddings, we show that it can provide an easy way to
adapt embeddings to specific tasks.
