  The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing
with a focus on semantic parsing and text generation. Currently, we witness an
excellent performance of neural parsers and generators on the PMB. This might
suggest that such semantic processing tasks have by and large been solved. We
argue that this is not the case and that performance scores from the past on
the PMB are inflated by non-optimal data splits and test sets that are too
easy. In response, we introduce several changes. First, instead of the prior
random split, we propose a more systematic splitting approach to improve the
reliability of the standard test data. Second, except for the standard test
set, we also propose two challenge sets: one with longer texts including
discourse structure, and one that addresses compositional generalization. We
evaluate five neural models for semantic parsing and meaning-to-text
generation. Our results show that model performance declines (in some cases
dramatically) on the challenge sets, revealing the limitations of neural models
when confronting such challenges.
