  Self-supervision based on the information extracted from large knowledge
graphs has been shown to improve the generalization of language models, in
zero-shot evaluation on various downstream language reasoning tasks. Since
these improvements are reported in aggregate, however, little is known about
(i) how to select the appropriate knowledge for solid performance across tasks,
(ii) how to combine this knowledge with neural language models, and (iii) how
these pairings affect granular task performance. In this paper, we study the
effect of knowledge sampling strategies and sizes that can be used to generate
synthetic data for adapting language models. We study the effect of different
synthetic datasets on language models with various architectures and sizes. The
resulting models are evaluated against four task properties: domain overlap,
answer similarity, vocabulary overlap, and answer length. Our experiments show
that encoder-decoder models benefit from more data to learn from, whereas
sampling strategies that balance across different aspects yield best
performance. Most of the improvement occurs on questions with short answers and
dissimilar answer candidates, which corresponds to the characteristics of the
data used for pre-training.
