  Several metrics have been proposed for assessing the similarity of (abstract)
meaning representations (AMRs), but little is known about how they relate to
human similarity ratings. Moreover, the current metrics have complementary
strengths and weaknesses: some emphasize speed, while others make the alignment
of graph structures explicit, at the price of a costly alignment step.
  In this work we propose new Weisfeiler-Leman AMR similarity metrics that
unify the strengths of previous metrics, while mitigating their weaknesses.
Specifically, our new metrics are able to match contextualized substructures
and induce n:m alignments between their nodes. Furthermore, we introduce a
Benchmark for AMR Metrics based on Overt Objectives (BAMBOO), the first
benchmark to support empirical assessment of graph-based MR similarity metrics.
BAMBOO maximizes the interpretability of results by defining multiple overt
objectives that range from sentence similarity objectives to stress tests that
probe a metric's robustness against meaning-altering and meaning-preserving
graph transformations. We show the benefits of BAMBOO by profiling previous
metrics and our own metrics. Results indicate that our novel metrics may serve
as a strong baseline for future work.
