  Code generation systems have been extensively developed in recent years to
generate source code based on natural language instructions. However, despite
their advancements, these systems still face robustness issues where even
slightly different instructions can result in significantly different code
semantics. Robustness is critical for code generation systems, as it can have
significant impacts on software development, software quality, and trust in the
generated code. Although existing testing techniques for general text-to-text
software can detect some robustness issues, they are limited in effectiveness
due to ignoring the characteristics of code generation systems. In this work,
we propose a novel technique COCO to test the robustness of code generation
systems. It exploits the usage scenario of code generation systems to make the
original programming instruction more concrete by incorporating features known
to be contained in the original code. A robust system should maintain code
semantics for the concretized instruction, and COCO detects robustness
inconsistencies when it does not. We evaluated COCO on eight advanced code
generation systems, including commercial tools such as Copilot and ChatGPT,
using two widely-used datasets. Our results demonstrate the effectiveness of
COCO in testing the robustness of code generation systems, outperforming two
techniques adopted from general text-to-text software testing by 466.66% and
104.02%, respectively. Furthermore, concretized instructions generated by COCO
can help reduce robustness inconsistencies by 18.35% to 53.91% through
fine-tuning.
