  We consider the intrinsic evaluation of neural generative dialog models
through the lens of Grice's Maxims of Conversation (1975). Based on the maxim
of Quantity (be informative), we propose Relative Utterance Quantity (RUQ) to
diagnose the `I don't know' problem, in which a dialog system produces generic
responses. The linguistically motivated RUQ diagnostic compares the model score
of a generic response to that of the reference response. We find that for
reasonable baseline models, `I don't know' is preferred over the reference the
majority of the time, but this can be reduced to less than 5% with
hyperparameter tuning. RUQ allows for the direct analysis of the `I don't know'
problem, which has been addressed but not analyzed by prior work.
