  English Natural Language Understanding (NLU) systems have achieved great
performances and even outperformed humans on benchmarks like GLUE and
SuperGLUE. However, these benchmarks contain only textbook Standard American
English (SAE). Other dialects have been largely overlooked in the NLP
community. This leads to biased and inequitable NLU systems that serve only a
sub-population of speakers. To understand disparities in current models and to
facilitate more dialect-competent NLU systems, we introduce the VernAcular
Language Understanding Evaluation (VALUE) benchmark, a challenging variant of
GLUE that we created with a set of lexical and morphosyntactic transformation
rules. In this initial release (V.1), we construct rules for 11 features of
African American Vernacular English (AAVE), and we recruit fluent AAVE speakers
to validate each feature transformation via linguistic acceptability judgments
in a participatory design manner. Experiments show that these new dialectal
features can lead to a drop in model performance. To run the transformation
code and download both synthetic and gold-standard dialectal GLUE benchmarks,
see https://github.com/SALT-NLP/value
