  Over-parameterized models, typically pretrained language models (LMs), have
shown an appealing expressive power due to their small learning bias. However,
the huge learning capacity of LMs can also lead to large learning variance. In
a pilot study, we find that, when faced with multiple domains, a critical
portion of parameters behave unexpectedly in a domain-specific manner while
others behave in a domain-general one. Motivated by this phenomenon, we for the
first time posit that domain-general parameters can underpin a domain-general
LM that can be derived from the original LM. To uncover the domain-general LM,
we propose to identify domain-general parameters by playing lottery tickets
(dubbed doge tickets). In order to intervene the lottery, we propose a
domain-general score, which depicts how domain-invariant a parameter is by
associating it with the variance. Comprehensive experiments are conducted on
the Amazon, Mnli and OntoNotes datasets. The results show that the doge tickets
obtains an improved out-of-domain generalization in comparison with a range of
competitive baselines. Analysis results further hint the existence of
domain-general parameters and the performance consistency of doge tickets.
