  The logical negation property (LNP), which implies generating different
predictions for semantically opposite inputs, is an important property that a
trustworthy language model must satisfy. However, much recent evidence shows
that large-size pre-trained language models (PLMs) do not satisfy this
property. In this paper, we perform experiments using probing tasks to assess
PLM's LNP understanding. Unlike previous studies that only examined negation
expressions, we expand the boundary of the investigation to lexical semantics.
Through experiments, we observe that PLMs violate the LNP frequently. To
alleviate the issue, we propose a novel intermediate training task, names
meaning-matching, designed to directly learn a meaning-text correspondence,
instead of relying on the distributional hypothesis. Through multiple
experiments, we find that the task enables PLMs to learn lexical semantic
information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm
that it is a safe intermediate task that guarantees a similar or better
performance of downstream tasks. Finally, we observe that our proposed approach
outperforms our previous counterparts despite its time and resource efficiency.
