  Unsupervised cross-lingual transfer involves transferring knowledge between
languages without explicit supervision. Although numerous studies have been
conducted to improve performance in such tasks by focusing on cross-lingual
knowledge, particularly lexical and syntactic knowledge, current approaches are
limited as they only incorporate syntactic or lexical information. Since each
type of information offers unique advantages and no previous attempts have
combined both, we attempt to explore the potential of this approach. In this
paper, we present a novel framework called "Lexicon-Syntax Enhanced
Multilingual BERT" that combines both lexical and syntactic knowledge.
Specifically, we use Multilingual BERT (mBERT) as the base model and employ two
techniques to enhance its learning capabilities. The code-switching technique
is used to implicitly teach the model lexical alignment information, while a
syntactic-based graph attention network is designed to help the model encode
syntactic structure. To integrate both types of knowledge, we input
code-switched sequences into both the syntactic module and the mBERT base model
simultaneously. Our extensive experimental results demonstrate this framework
can consistently outperform all baselines of zero-shot cross-lingual transfer,
with the gains of 1.0~3.7 points on text classification, named entity
recognition (ner), and semantic parsing tasks. Keywords:cross-lingual transfer,
lexicon, syntax, code-switching, graph attention network
