  Testing of machine learning (ML) models is a known challenge identified by
researchers and practitioners alike. Unfortunately, current practice for ML
model testing prioritizes testing for model performance, while often neglecting
the requirements and constraints of the ML-enabled system that integrates the
model. This limited view of testing leads to failures during integration,
deployment, and operations, contributing to the difficulties of moving models
from development to production. This paper presents an approach based on
quality attribute (QA) scenarios to elicit and define system- and
model-relevant test cases for ML models. The QA-based approach described in
this paper has been integrated into MLTE, a process and tool to support ML
model test and evaluation. Feedback from users of MLTE highlights its
effectiveness in testing beyond model performance and identifying failures
early in the development process.
