  Pre-trained language models (PTLM) have achieved impressive results in a
range of natural language understanding (NLU) and generation (NLG) tasks.
However, current pre-training objectives such as masked token prediction (for
BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not
explicitly model the relational commonsense knowledge about everyday concepts,
which is crucial to many downstream tasks that need common sense to understand
or generate. To augment PTLMs with concept-centric commonsense knowledge, in
this paper, we propose both generative and contrastive objectives for learning
common sense from the text, and use them as intermediate self-supervised
learning tasks for incrementally pre-training PTLMs (before task-specific
fine-tuning on downstream datasets). Furthermore, we develop a joint
pre-training framework to unify generative and contrastive objectives so that
they can mutually reinforce each other. Extensive experimental results show
that our method, concept-aware language model (CALM), can pack more commonsense
knowledge into the parameters of a pre-trained text-to-text transformer without
relying on external knowledge graphs, yielding better performance on both NLU
and NLG tasks. We show that while only incrementally pre-trained on a
relatively small corpus for a few steps, CALM outperforms baseline methods by a
consistent margin and even comparable with some larger PTLMs, which suggests
that CALM can serve as a general, plug-and-play method for improving the
commonsense reasoning ability of a PTLM.
