  Recent work in cross-lingual contextual word embedding learning cannot handle
multi-sense words well. In this work, we explore the characteristics of
contextual word embeddings and show the link between contextual word embeddings
and word senses. We propose two improving solutions by considering contextual
multi-sense word embeddings as noise (removal) and by generating cluster level
average anchor embeddings for contextual multi-sense word embeddings
(replacement). Experiments show that our solutions can improve the supervised
contextual word embeddings alignment for multi-sense words in a microscopic
perspective without hurting the macroscopic performance on the bilingual
lexicon induction task. For unsupervised alignment, our methods significantly
improve the performance on the bilingual lexicon induction task for more than
10 points.
