  The life cycle of machine learning (ML) applications consists of two stages:
model development and model deployment. However, traditional ML systems (e.g.,
training-specific or inference-specific systems) focus on one particular stage
or phase of the life cycle of ML applications. These systems often aim at
optimizing model training or accelerating model inference, and they frequently
assume homogeneous infrastructure, which may not always reflect real-world
scenarios that include cloud data centers, local servers, containers, and
serverless platforms. We present StraightLine, an end-to-end resource-aware
scheduler that schedules the optimal resources (e.g., container, virtual
machine, or serverless) for different ML application requests in a hybrid
infrastructure. The key innovation is an empirical dynamic placing algorithm
that intelligently places requests based on their unique characteristics (e.g.,
request frequency, input data size, and data distribution). In contrast to
existing ML systems, StraightLine offers end-to-end resource-aware placement,
thereby it can significantly reduce response time and failure rate for model
deployment when facing different computing resources in the hybrid
infrastructure.
