  Transformer architectures have achieved great success in solving natural
language tasks, which learn strong language representations from large-scale
unlabeled texts. In this paper, we seek to go further beyond and explore a new
logical inductive bias for better language representation learning. Logic
reasoning is known as a formal methodology to reach answers from given
knowledge and facts. Inspired by such a view, we develop a novel neural
architecture named FOLNet (First-Order Logic Network), to encode this new
inductive bias. We construct a set of neural logic operators as learnable Horn
clauses, which are further forward-chained into a fully differentiable neural
architecture (FOLNet). Interestingly, we find that the self-attention module in
transformers can be composed by two of our neural logic operators, which
probably explains their strong reasoning performance. Our proposed FOLNet has
the same input and output interfaces as other pretrained models and thus could
be pretrained/finetuned by using similar losses. It also allows FOLNet to be
used in a plug-and-play manner when replacing other pretrained models. With our
logical inductive bias, the same set of ``logic deduction skills'' learned
through pretraining are expected to be equally capable of solving diverse
downstream tasks. For this reason, FOLNet learns language representations that
have much stronger transfer capabilities. Experimental results on several
language understanding tasks show that our pretrained FOLNet model outperforms
the existing strong transformer-based approaches.
