  Neural approaches to Natural Language Generation (NLG) have been promising
for goal-oriented dialogue. One of the challenges of productionizing these
approaches, however, is the ability to control response quality, and ensure
that generated responses are acceptable. We propose the use of a generate,
filter, and rank framework, in which candidate responses are first filtered to
eliminate unacceptable responses, and then ranked to select the best response.
While acceptability includes grammatical correctness and semantic correctness,
we focus only on grammaticality classification in this paper, and show that
existing datasets for grammatical error correction don't correctly capture the
distribution of errors that data-driven generators are likely to make. We
release a grammatical classification and semantic correctness classification
dataset for the weather domain that consists of responses generated by 3
data-driven NLG systems. We then explore two supervised learning approaches
(CNNs and GBDTs) for classifying grammaticality. Our experiments show that
grammaticality classification is very sensitive to the distribution of errors
in the data, and that these distributions vary significantly with both the
source of the response as well as the domain. We show that it's possible to
achieve high precision with reasonable recall on our dataset.
