  The robustness of Text-to-SQL parsers against adversarial perturbations plays
a crucial role in delivering highly reliable applications. Previous studies
along this line primarily focused on perturbations in the natural language
question side, neglecting the variability of tables. Motivated by this, we
propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm to
measure the robustness of Text-to-SQL models. Following this proposition, we
curate ADVETA, the first robustness evaluation benchmark featuring natural and
realistic ATPs. All tested state-of-the-art models experience dramatic
performance drops on ADVETA, revealing models' vulnerability in real-world
practices. To defend against ATP, we build a systematic adversarial training
example generation framework tailored for better contextualization of tabular
data. Experiments show that our approach not only brings the best robustness
improvement against table-side perturbations but also substantially empowers
models against NL-side perturbations. We release our benchmark and code at:
https://github.com/microsoft/ContextualSP.
