  As an increasing number of businesses becomes powered by machine-learning,
inference becomes a core operation, with a growing trend to be offered as a
service. In this context, the inference task must meet certain service-level
objectives (SLOs), such as high throughput and low latency. However, these
targets can be compromised by interference caused by long- or short-lived
co-located tasks. Prior works focus on the generic problem of co-scheduling to
mitigate the effect of interference on the performance-critical task. In this
work, we focus on inference pipelines and propose ODIN, a technique to mitigate
the effect of interference on the performance of the inference task, based on
the online scheduling of the pipeline stages. Our technique detects
interference online and automatically re-balances the pipeline stages to
mitigate the performance degradation of the inference task. We demonstrate that
ODIN successfully mitigates the effect of interference, sustaining the latency
and throughput of CNN inference, and outperforms the least-loaded scheduling
(LLS), a common technique for interference mitigation. Additionally, it is
effective in maintaining service-level objectives for inference, and it is
scalable to large network models executing on multiple processing elements.
