  We consider the question: what is the abstraction that should be implemented
by the computational engine of a machine learning system? Current machine
learning systems typically push whole tensors through a series of compute
kernels such as matrix multiplications or activation functions, where each
kernel runs on an AI accelerator (ASIC) such as a GPU. This implementation
abstraction provides little built-in support for ML systems to scale past a
single machine, or for handling large models with matrices or tensors that do
not easily fit into the RAM of an ASIC. In this paper, we present an
alternative implementation abstraction called the tensor relational algebra
(TRA). The TRA is a set-based algebra based on the relational algebra.
Expressions in the TRA operate over binary tensor relations, where keys are
multi-dimensional arrays and values are tensors. The TRA is easily executed
with high efficiency in a parallel or distributed environment, and amenable to
automatic optimization. Our empirical study shows that the optimized TRA-based
back-end can significantly outperform alternatives for running ML workflows in
distributed clusters.
