  Transformer-based pre-trained models of code (PTMC) have been widely utilized
and have achieved state-of-the-art performance in many mission-critical
applications. However, they can be vulnerable to adversarial attacks through
identifier substitution or coding style transformation, which can significantly
degrade accuracy and may further incur security concerns. Although several
approaches have been proposed to generate adversarial examples for PTMC, the
effectiveness and efficiency of such approaches, especially on different code
intelligence tasks, has not been well understood. To bridge this gap, this
study systematically analyzes five state-of-the-art adversarial attack
approaches from three perspectives: effectiveness, efficiency, and the quality
of generated examples. The results show that none of the five approaches
balances all these perspectives. Particularly, approaches with a high attack
success rate tend to be time-consuming; the adversarial code they generate
often lack naturalness, and vice versa. To address this limitation, we explore
the impact of perturbing identifiers under different contexts and find that
identifier substitution within for and if statements is the most effective.
Based on these findings, we propose a new approach that prioritizes different
types of statements for various tasks and further utilizes beam search to
generate adversarial examples. Evaluation results show that it outperforms the
state-of-the-art ALERT in terms of both effectiveness and efficiency while
preserving the naturalness of the generated adversarial examples.
