  The advent of advanced AI underscores the urgent need for comprehensive
safety evaluations, necessitating collaboration across communities (i.e., AI,
software engineering, and governance). However, divergent practices and
terminologies across these communities, combined with the complexity of AI
systems-of which models are only a part-and environmental affordances (e.g.,
access to tools), obstruct effective communication and comprehensive
evaluation. This paper proposes a framework for AI system evaluation comprising
three components: 1) harmonised terminology to facilitate communication across
communities involved in AI safety evaluation; 2) a taxonomy identifying
essential elements for AI system evaluation; 3) a mapping between AI lifecycle,
stakeholders, and requisite evaluations for accountable AI supply chain. This
framework catalyses a deeper discourse on AI system evaluation beyond
model-centric approaches.
