  Semantic role labeling (SRL) identifies the predicate-argument structure in a
sentence. This task is usually accomplished in four steps: predicate
identification, predicate sense disambiguation, argument identification, and
argument classification. Errors introduced at one step propagate to later
steps. Unfortunately, the existing SRL evaluation scripts do not consider the
full effect of this error propagation aspect. They either evaluate arguments
independent of predicate sense (CoNLL09) or do not evaluate predicate sense at
all (CoNLL05), yielding an inaccurate SRL model performance on the argument
classification task. In this paper, we address key practical issues with
existing evaluation scripts and propose a more strict SRL evaluation metric
PriMeSRL. We observe that by employing PriMeSRL, the quality evaluation of all
SoTA SRL models drops significantly, and their relative rankings also change.
We also show that PriMeSRLsuccessfully penalizes actual failures in SoTA SRL
models.
