  Compound AI systems (CASs) that employ LLMs as agents to accomplish
knowledge-intensive tasks via interactions with tools and data retrievers have
garnered significant interest within database and AI communities. While these
systems have the potential to supplement typical analysis workflows of data
analysts in enterprise data platforms, unfortunately, CASs are subject to the
same data discovery challenges that analysts have encountered over the years --
silos of multimodal data sources, created across teams and departments within
an organization, make it difficult to identify appropriate data sources for
accomplishing the task at hand. Existing data discovery benchmarks do not model
such multimodality and multiplicity of data sources. Moreover, benchmarks of
CASs prioritize only evaluating end-to-end task performance. To catalyze
research on evaluating the data discovery performance of multimodal data
retrievers in CASs within a real-world setting, we propose CMDBench, a
benchmark modeling the complexity of enterprise data platforms. We adapt
existing datasets and benchmarks in open-domain -- from question answering and
complex reasoning tasks to natural language querying over structured data -- to
evaluate coarse- and fine-grained data discovery and task execution
performance. Our experiments reveal the impact of data retriever design on
downstream task performance -- a 46% drop in task accuracy on average -- across
various modalities, data sources, and task difficulty. The results indicate the
need to develop optimization strategies to identify appropriate LLM agents and
retrievers for efficient execution of CASs over enterprise data.
