  Recent advances in large language models have raised wide concern in
generating abundant plausible source code without scrutiny, and thus tracing
the provenance of code emerges as a critical issue. To solve the issue, we
propose CodeMark, a watermarking system that hides bit strings into variables
respecting the natural and operational semantics of the code. For naturalness,
we novelly introduce a contextual watermarking scheme to generate watermarked
variables more coherent in the context atop graph neural networks. Each
variable is treated as a node on the graph and the node feature gathers
neighborhood (context) information through learning. Watermarks embedded into
the features are thus reflected not only by the variables but also by the local
contexts. We further introduce a pre-trained model on source code as a teacher
to guide more natural variable generation. Throughout the embedding, the
operational semantics are preserved as only variable names are altered. Beyond
guaranteeing code-specific properties, CodeMark is superior in watermarking
accuracy, capacity, and efficiency due to a more diversified pattern generated.
Experimental results show CodeMark outperforms the SOTA watermarking systems
with a better balance of the watermarking requirements.
