  We propose a novel way to handle out of vocabulary (OOV) words in downstream
natural language processing (NLP) tasks. We implement a network that predicts
useful embeddings for OOV words based on their morphology and on the context in
which they appear. Our model also incorporates an attention mechanism
indicating the focus allocated to the left context words, the right context
words or the word's characters, hence making the prediction more interpretable.
The model is a ``drop-in'' module that is jointly trained with the downstream
task's neural network, thus producing embeddings specialized for the task at
hand. When the task is mostly syntactical, we observe that our model aims most
of its attention on surface form characters. On the other hand, for tasks more
semantical, the network allocates more attention to the surrounding words. In
all our tests, the module helps the network to achieve better performances in
comparison to the use of simple random embeddings.
