  On-device inference holds great potential for increased energy efficiency,
responsiveness, and privacy in edge ML systems. However, due to less capable ML
models that can be embedded in resource-limited devices, use cases are limited
to simple inference tasks such as visual keyword spotting, gesture recognition,
and predictive analytics. In this context, the Hierarchical Inference (HI)
system has emerged as a promising solution that augments the capabilities of
the local ML by offloading selected samples to an edge server or cloud for
remote ML inference. Existing works demonstrate through simulation that HI
improves accuracy. However, they do not account for the latency and energy
consumption on the device, nor do they consider three key heterogeneous
dimensions that characterize ML systems: hardware, network connectivity, and
models. In contrast, this paper systematically compares the performance of HI
with on-device inference based on measurements of accuracy, latency, and energy
for running embedded ML models on five devices with different capabilities and
three image classification datasets. For a given accuracy requirement, the HI
systems we designed achieved up to 73% lower latency and up to 77% lower device
energy consumption than an on-device inference system. The key to building an
efficient HI system is the availability of small-size, reasonably accurate
on-device models whose outputs can be effectively differentiated for samples
that require remote inference. Despite the performance gains, HI requires
on-device inference for all samples, which adds a fixed overhead to its latency
and energy consumption. Therefore, we design a hybrid system, Early Exit with
HI (EE-HI), and demonstrate that compared to HI, EE-HI reduces the latency by
up to 59.7% and lowers the device's energy consumption by up to 60.4%.
