  Today, using multiple heterogeneous accelerators efficiently from
applications and high-level frameworks, such as TensorFlow and Caffe, poses
significant challenges in three respects: (a) sharing accelerators, (b)
allocating available resources elastically during application execution, and
(c) reducing the required programming effort. In this paper, we present Arax, a
runtime system that decouples applications from heterogeneous accelerators
within a server. First, Arax maps application tasks dynamically to available
resources, managing all required task state, memory allocations, and task
dependencies. As a result, Arax can share accelerators across applications in a
server and adjust the resources used by each application as load fluctuates
over time. dditionally, Arax offers a simple API and includes Autotalk, a stub
generator that automatically generates stub libraries for applications already
written for specific accelerator types, such as NVIDIA GPUs. Consequently, Arax
applications are written once without considering physical details, including
the number and type of accelerators. Our results show that applications, such
as Caffe, TensorFlow, and Rodinia, can run using Arax with minimum effort and
low overhead compared to native execution, about 12% (geometric mean). Arax
supports efficient accelerator sharing, by offering up to 20% improved
execution times compared to NVIDIA MPS, which supports NVIDIA GPUs only. Arax
can transparently provide elasticity, decreasing total application turn-around
time by up to 2x compared to native execution without elasticity support.
