  Resource-constrained Edge Devices (EDs), e.g., IoT sensors and
microcontroller units, are expected to make intelligent decisions using Deep
Learning (DL) inference at the edge of the network. Toward this end, there is a
significant research effort in developing tinyML models - Deep Learning (DL)
models with reduced computation and memory storage requirements - that can be
embedded on these devices. However, tinyML models have lower inference
accuracy. On a different front, DNN partitioning and inference offloading
techniques were studied for distributed DL inference between EDs and Edge
Servers (ESs). In this paper, we explore Hierarchical Inference (HI), a novel
approach proposed by Vishnu et al. 2023, arXiv:2304.00891v1 , for performing
distributed DL inference at the edge. Under HI, for each data sample, an ED
first uses a local algorithm (e.g., a tinyML model) for inference. Depending on
the application, if the inference provided by the local algorithm is incorrect
or further assistance is required from large DL models on edge or cloud, only
then the ED offloads the data sample. At the outset, HI seems infeasible as the
ED, in general, cannot know if the local inference is sufficient or not.
Nevertheless, we present the feasibility of implementing HI for machine fault
detection and image classification applications. We demonstrate its benefits
using quantitative analysis and argue that using HI will result in low latency,
bandwidth savings, and energy savings in edge AI systems.
