  In this paper, we introduce UniBridge (Cross-Lingual Transfer Learning with
Optimized Embeddings and Vocabulary), a comprehensive approach developed to
improve the effectiveness of Cross-Lingual Transfer Learning, particularly in
languages with limited resources. Our approach tackles two essential elements
of a language model: the initialization of embeddings and the optimal
vocabulary size. Specifically, we propose a novel embedding initialization
method that leverages both lexical and semantic alignment for a language. In
addition, we present a method for systematically searching for the optimal
vocabulary size, ensuring a balance between model complexity and linguistic
coverage. Our experiments across multilingual datasets show that our approach
greatly improves the F1-Score in several languages. UniBridge is a robust and
adaptable solution for cross-lingual systems in various languages, highlighting
the significance of initializing embeddings and choosing the right vocabulary
size in cross-lingual environments.
