  The growth of social media has encouraged the written use of African American
Vernacular English (AAVE), which has traditionally been used only in oral
contexts. However, NLP models have historically been developed using dominant
English varieties, such as Standard American English (SAE), due to text corpora
availability. We investigate the performance of GPT-2 on AAVE text by creating
a dataset of intent-equivalent parallel AAVE/SAE tweet pairs, thereby isolating
syntactic structure and AAVE- or SAE-specific language for each pair. We
evaluate each sample and its GPT-2 generated text with pretrained sentiment
classifiers and find that while AAVE text results in more classifications of
negative sentiment than SAE, the use of GPT-2 generally increases occurrences
of positive sentiment for both. Additionally, we conduct human evaluation of
AAVE and SAE text generated with GPT-2 to compare contextual rigor and overall
quality.
