  Current trends in Machine Learning~(ML) inference on hardware accelerated
devices (e.g., GPUs, TPUs) point to alarmingly low utilization. As ML inference
is increasingly time-bounded by tight latency SLOs, increasing data parallelism
is not an option. The need for better efficiency motivates GPU multiplexing.
Furthermore, existing GPU programming abstractions force programmers to
micro-manage GPU resources in an early-binding, context-free fashion. We
propose a VLIW-inspired Out-of-Order (OoO) Just-in-Time (JIT) compiler that
coalesces and reorders execution kernels at runtime for throughput-optimal
device utilization while satisfying latency SLOs. We quantify the
inefficiencies of space-only and time-only multiplexing alternatives and
demonstrate an achievable 7.7x opportunity gap through spatial coalescing.
