  There has been much interest in deploying deep learning algorithms on
low-powered devices, including smartphones, drones, and medical sensors.
However, full-scale deep neural networks are often too resource-intensive in
terms of energy and storage. As a result, the bulk part of the machine learning
operation is therefore often carried out on an edge server, where the data is
compressed and transmitted. However, compressing data (such as images) leads to
transmitting information irrelevant to the supervised task. Another popular
approach is to split the deep network between the device and the server while
compressing intermediate features. To date, however, such split computing
strategies have barely outperformed the aforementioned naive data compression
baselines due to their inefficient approaches to feature compression. This
paper adopts ideas from knowledge distillation and neural image compression to
compress intermediate feature representations more efficiently. Our supervised
compression approach uses a teacher model and a student model with a stochastic
bottleneck and learnable prior for entropy coding (Entropic Student). We
compare our approach to various neural image and feature compression baselines
in three vision tasks and found that it achieves better supervised
rate-distortion performance while maintaining smaller end-to-end latency. We
furthermore show that the learned feature representations can be tuned to serve
multiple downstream tasks.
