  Large Language Models (LLMs) have showcased remarkable capabilities
surpassing conventional NLP challenges, creating opportunities for use in
production use cases. Towards this goal, there is a notable shift to building
compound AI systems, wherein LLMs are integrated into an expansive software
infrastructure with many components like models, retrievers, databases and
tools. In this paper, we introduce a blueprint architecture for compound AI
systems to operate in enterprise settings cost-effectively and feasibly. Our
proposed architecture aims for seamless integration with existing compute and
data infrastructure, with ``stream'' serving as the key orchestration concept
to coordinate data and instructions among agents and other components. Task and
data planners, respectively, break down, map, and optimize tasks and data to
available agents and data sources defined in respective registries, given
production constraints such as accuracy and latency.
