  Deployed multimodal systems can fail in ways that evaluators did not
anticipate. In order to find these failures before deployment, we introduce
MultiMon, a system that automatically identifies systematic failures --
generalizable, natural-language descriptions of patterns of model failures. To
uncover systematic failures, MultiMon scrapes a corpus for examples of
erroneous agreement: inputs that produce the same output, but should not. It
then prompts a language model (e.g., GPT-4) to find systematic patterns of
failure and describe them in natural language. We use MultiMon to find 14
systematic failures (e.g., "ignores quantifiers") of the CLIP text-encoder,
each comprising hundreds of distinct inputs (e.g., "a shelf with a few/many
books"). Because CLIP is the backbone for most state-of-the-art multimodal
systems, these inputs produce failures in Midjourney 5.1, DALL-E, VideoFusion,
and others. MultiMon can also steer towards failures relevant to specific use
cases, such as self-driving cars. We see MultiMon as a step towards evaluation
that autonomously explores the long tail of potential system failures. Code for
MULTIMON is available at https://github.com/tsb0601/MultiMon.
