  Large pretrained multilingual language models (ML-LMs) have shown remarkable
capabilities of zero-shot cross-lingual transfer, without direct cross-lingual
supervision. While these results are promising, follow-up works found that,
within the multilingual embedding spaces, there exists strong language identity
information which hinders the expression of linguistic factors shared across
languages. For semantic tasks like cross-lingual sentence retrieval, it is
desired to remove such language identity signals to fully leverage semantic
information. In this work, we provide a novel view of projecting away
language-specific factors from a multilingual embedding space. Specifically, we
discover that there exists a low-rank subspace that primarily encodes
information irrelevant to semantics (e.g., syntactic information). To identify
this subspace, we present a simple but effective unsupervised method based on
singular value decomposition with multiple monolingual corpora as input. Once
the subspace is found, we can directly project the original embeddings into the
null space to boost language agnosticism without finetuning. We systematically
evaluate our method on various tasks including the challenging
language-agnostic QA retrieval task. Empirical results show that applying our
method consistently leads to improvements over commonly used ML-LMs.
