  Deep Learning (DL) models to analyze source code have shown immense promise
during the past few years. More recently, self-supervised pre-training has
gained traction for learning generic code representations valuable for many
downstream SE tasks, such as clone and bug detection.
  While previous work successfully learned from different code abstractions
(e.g., token, AST, graph), we argue that it is also essential to factor in how
developers code day-to-day for general-purpose representation learning. On the
one hand, human developers tend to write repetitive programs referencing
existing code snippets from the current codebase or online resources (e.g.,
Stack Overflow website) rather than implementing functions from scratch; such
behaviors result in a vast number of code clones. In contrast, a deviant clone
by mistake might trigger malicious program behaviors.
  Thus, as a proxy to incorporate developers' coding behavior into the
pre-training scheme, we propose to include code clones and their deviants. In
particular, we propose CONCORD, a self-supervised, contrastive learning
strategy to place benign clones closer in the representation space while moving
deviants further apart. We show that CONCORD's clone-aware contrastive learning
drastically reduces the need for expensive pre-training resources while
improving the performance of downstream SE tasks. We also empirically
demonstrate that CONCORD can improve existing pre-trained models to learn
better representations that consequently become more efficient in both
identifying semantically equivalent programs and differentiating buggy from
non-buggy code.
