  With the success of neural language models (LMs), their language acquisition
has gained much attention. This work sheds light on the second language (L2)
acquisition of LMs, while previous work has typically explored their first
language (L1) acquisition. Specifically, we trained bilingual LMs with a
scenario similar to human L2 acquisition and analyzed their cross-lingual
transfer from linguistic perspectives. Our exploratory experiments demonstrated
that the L1 pretraining accelerated their linguistic generalization in L2, and
language transfer configurations (e.g., the L1 choice, and presence of parallel
texts) substantially affected their generalizations. These clarify their
(non-)human-like L2 acquisition in particular aspects.
