  When deploying deep learning models to a device, it is traditionally assumed
that available computational resources (compute, memory, and power) remain
static. However, real-world computing systems do not always provide stable
resource guarantees. Computational resources need to be conserved when load
from other processes is high or battery power is low. Inspired by recent works
on neural network subspaces, we propose a method for training a "compressible
subspace" of neural networks that contains a fine-grained spectrum of models
that range from highly efficient to highly accurate. Our models require no
retraining, thus our subspace of models can be deployed entirely on-device to
allow adaptive network compression at inference time. We present results for
achieving arbitrarily fine-grained accuracy-efficiency trade-offs at inference
time for structured and unstructured sparsity. We achieve accuracies on-par
with standard models when testing our uncompressed models, and maintain high
accuracy for sparsity rates above 90% when testing our compressed models. We
also demonstrate that our algorithm extends to quantization at variable bit
widths, achieving accuracy on par with individually trained networks.
