  Benchmarks for large multimodal language models (MLMs) now serve to
simultaneously assess the general capabilities of models instead of evaluating
for a specific capability. As a result, when a developer wants to identify
which models to use for their application, they are overwhelmed by the number
of benchmarks and remain uncertain about which benchmark's results are most
reflective of their specific use case. This paper introduces Task-Me-Anything,
a benchmark generation engine which produces a benchmark tailored to a user's
needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and
can programmatically generate a vast number of task instances. Additionally, it
algorithmically addresses user queries regarding MLM performance efficiently
within a computational budget. It contains 113K images, 10K videos, 2K 3D
object assets, over 365 object categories, 655 attributes, and 335
relationships. It can generate 750M image/video question-answering pairs, which
focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals
critical insights: open-source MLMs excel in object and attribute recognition
but lack spatial and temporal understanding; each model exhibits unique
strengths and weaknesses; larger models generally perform better, though
exceptions exist; and GPT4o demonstrates challenges in recognizing
rotating/moving objects and distinguishing colors.
