  Word embeddings are an essential component in a wide range of natural
language processing applications. However, distributional semantic models are
known to struggle when only a small number of context sentences are available.
Several methods have been proposed to obtain higher-quality vectors for these
words, leveraging both this context information and sometimes the word forms
themselves through a hybrid approach. We show that the current tasks do not
suffice to evaluate models that use word-form information, as such models can
easily leverage word forms in the training data that are related to word forms
in the test data. We introduce 3 new tasks, allowing for a more balanced
comparison between models. Furthermore, we show that hyperparameters that have
largely been ignored in previous work can consistently improve the performance
of both baseline and advanced models, achieving a new state of the art on 4 out
of 6 tasks.
