  This technical report briefly describes our JDExplore d-team's Vega v2
submission on the SuperGLUE leaderboard. SuperGLUE is more challenging than the
widely used general language understanding evaluation (GLUE) benchmark,
containing eight difficult language understanding tasks, including question
answering, natural language inference, word sense disambiguation, coreference
resolution, and reasoning. [Method] Instead of arbitrarily increasing the size
of a pretrained language model (PLM), our aim is to 1) fully extract knowledge
from the input pretraining data given a certain parameter budget, e.g., 6B, and
2) effectively transfer this knowledge to downstream tasks. To achieve goal 1),
we propose self-evolution learning for PLMs to wisely predict the informative
tokens that should be masked, and supervise the masked language modeling (MLM)
process with rectified smooth labels. For goal 2), we leverage the prompt
transfer technique to improve the low-resource tasks by transferring the
knowledge from the foundation model and related downstream tasks to the target
task. [Results] According to our submission record (Oct. 2022), with our
optimized pretraining and fine-tuning strategies, our 6B Vega method achieved
new state-of-the-art performance on 4/8 tasks, sitting atop the SuperGLUE
leaderboard on Oct. 8, 2022, with an average score of 91.3.
