  Verbs play a critical role in the meaning of sentences, but these ubiquitous
words have received little attention in recent distributional semantics
research. We introduce SimVerb-3500, an evaluation resource that provides human
ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed
verb types from the USF free-association database, providing at least three
examples for every VerbNet class. This broad coverage facilitates detailed
analyses of how syntactic and semantic phenomena together influence human
understanding of verb meaning. Further, with significantly larger development
and test sets than existing benchmarks, SimVerb-3500 enables more robust
evaluation of representation learning architectures and promotes the
development of methods tailored to verbs. We hope that SimVerb-3500 will enable
a richer understanding of the diversity and complexity of verb semantics and
guide the development of systems that can effectively represent and interpret
this meaning.
