  Metaphor identification aims at understanding whether a given expression is
used figuratively in context. However, in this paper we show how existing
metaphor identification datasets can be gamed by fully ignoring the potential
metaphorical expression or the context in which it occurs. We test this
hypothesis in a variety of datasets and settings, and show that metaphor
identification systems based on language models without complete information
can be competitive with those using the full context. This is due to the
construction procedures to build such datasets, which introduce unwanted biases
for positive and negative classes. Finally, we test the same hypothesis on
datasets that are carefully sampled from natural corpora and where this bias is
not present, making these datasets more challenging and reliable.
