  Despite exciting progress in causal language models, the expressiveness of
the representations is largely limited due to poor discrimination ability. To
remedy this issue, we present ContraCLM, a novel contrastive learning framework
at both token-level and sequence-level. We assess ContraCLM on a variety of
downstream tasks. We show that ContraCLM enhances discrimination of the
representations and bridges the gap with the encoder-only models, which makes
causal language models better suited for tasks beyond language generation.
Specifically, we attain $44\%$ relative improvement on the Semantic Textual
Similarity tasks and $34\%$ on Code-to-Code Search tasks. Furthermore, by
improving the expressiveness of the representations, ContraCLM also boosts the
source code generation capability with $9\%$ relative improvement on execution
accuracy on the HumanEval benchmark.
