  Software is constantly changing, requiring developers to perform several
derived tasks in a timely manner, such as writing a description for the
intention of the code change, or identifying the defect-prone code changes.
Considering that the cost of dealing with these tasks can account for a large
proportion (typically around 70 percent) of the total development expenditure,
automating such processes will significantly lighten the burdens of developers.
To achieve such a target, existing approaches mainly rely on training deep
learning models from scratch or fine-tuning existing pretrained models on such
tasks, both of which have weaknesses. Specifically, the former uses
comparatively small-scale labelled data for training, making it difficult to
learn and exploit the domain knowledge of programming language hidden in the
large-amount unlabelled code in the wild; the latter is hard to fully leverage
the learned knowledge of the pre-trained model, as existing pre-trained models
are designed to encode a single code snippet rather than a code change (i.e.,
the difference between two code snippets). We propose to pre-train a model
specially designed for code changes to better support developers in software
maintenance. To this end, we first collect a large-scale dataset containing
1.5M+ pairwise data of code changes and commit messages. Based on these data,
we curate five different tasks for pre-training, which equip the model with
diverse domain knowledge about code changes. We fine-tune the pre-trained
model, CCT5, on three widely-studied tasks incurred by code changes and two
tasks specific to the code review process. Results show that CCT5 outperforms
both conventional deep learning approaches and existing pre-trained models on
these tasks.
