  Traditional neural word embeddings are usually dependent on a richer
diversity of vocabulary. However, the language models recline to cover major
vocabularies via the word embedding parameters, in particular, for multilingual
language models that generally cover a significant part of their overall
learning parameters. In this work, we present a new compact embedding structure
to reduce the memory footprint of the pre-trained language models with a
sacrifice of up to 4% absolute accuracy. The embeddings vectors reconstruction
follows a set of subspace embeddings and an assignment procedure via the
contextual relationship among tokens from pre-trained language models. The
subspace embedding structure calibrates to masked language models, to evaluate
our compact embedding structure on similarity and textual entailment tasks,
sentence and paraphrase tasks. Our experimental evaluation shows that the
subspace embeddings achieve compression rates beyond 99.8% in comparison with
the original embeddings for the language models on XNLI and GLUE benchmark
suites.
