  As AI applications for mobile devices become more prevalent, there is an
increasing need for faster execution and lower energy consumption for neural
model inference. Historically, the models run on mobile devices have been
smaller and simpler in comparison to large state-of-the-art research models,
which can only run on the cloud. However, cloud-only inference has drawbacks
such as increased network bandwidth consumption and higher latency. In
addition, cloud-only inference requires the input data (images, audio) to be
fully transferred to the cloud, creating concerns about potential privacy
breaches. We demonstrate an alternative approach: shared mobile-cloud
inference. Partial inference is performed on the mobile in order to reduce the
dimensionality of the input data and arrive at a compact feature tensor, which
is a latent space representation of the input signal. The feature tensor is
then transmitted to the server for further inference. This strategy can improve
inference latency, energy consumption, and network bandwidth usage, as well as
provide privacy protection, because the original signal never leaves the
mobile. Further performance gain can be achieved by compressing the feature
tensor before its transmission.
