  Recent advances in zero-shot and few-shot learning have shown promise for a
scope of research and practical purposes. However, this fast-growing area lacks
standardized evaluation suites for non-English languages, hindering progress
outside the Anglo-centric paradigm. To address this line of research, we
propose TAPE (Text Attack and Perturbation Evaluation), a novel benchmark that
includes six more complex NLU tasks for Russian, covering multi-hop reasoning,
ethical concepts, logic and commonsense knowledge. The TAPE's design focuses on
systematic zero-shot and few-shot NLU evaluation: (i) linguistic-oriented
adversarial attacks and perturbations for analyzing robustness, and (ii)
subpopulations for nuanced interpretation. The detailed analysis of testing the
autoregressive baselines indicates that simple spelling-based perturbations
affect the performance the most, while paraphrasing the input has a more
negligible effect. At the same time, the results demonstrate a significant gap
between the neural and human baselines for most tasks. We publicly release TAPE
(tape-benchmark.com) to foster research on robust LMs that can generalize to
new tasks when little to no supervision is available.
