  In the last year alone, a surge of new benchmarks to measure compositional
understanding of vision-language models have permeated the machine learning
ecosystem. Given an image, these benchmarks probe a model's ability to identify
its associated caption amongst a set of compositional distractors.
Surprisingly, we find significant biases in all these benchmarks rendering them
hackable. This hackability is so dire that blind models with no access to the
image outperform state-of-the-art vision-language models. To remedy this
rampant vulnerability, we introduce SugarCrepe, a new benchmark for
vision-language compositionality evaluation. We employ large language models,
instead of rule-based templates used in previous benchmarks, to generate fluent
and sensical hard negatives, and utilize an adversarial refinement mechanism to
maximally reduce biases. We re-evaluate state-of-the-art models and recently
proposed compositionality inducing strategies, and find that their improvements
were hugely overestimated, suggesting that more innovation is needed in this
important direction. We release SugarCrepe and the code for evaluation at:
https://github.com/RAIVNLab/sugar-crepe.
