  Benchmarks are crucial to measuring and steering progress in artificial
intelligence (AI). However, recent studies raised concerns over the state of AI
benchmarking, reporting issues such as benchmark overfitting, benchmark
saturation and increasing centralization of benchmark dataset creation. To
facilitate monitoring of the health of the AI benchmarking ecosystem, we
introduce methodologies for creating condensed maps of the global dynamics of
benchmark creation and saturation. We curated data for 3765 benchmarks covering
the entire domains of computer vision and natural language processing, and show
that a large fraction of benchmarks quickly trended towards near-saturation,
that many benchmarks fail to find widespread utilization, and that benchmark
performance gains for different AI tasks were prone to unforeseen bursts. We
analyze attributes associated with benchmark popularity, and conclude that
future benchmarks should emphasize versatility, breadth and real-world utility.
