  The trend of modeless ML inference is increasingly growing in popularity as
it hides the complexity of model inference from users and caters to diverse
user and application accuracy requirements. Previous work mostly focuses on
modeless inference in data centers. To provide low-latency inference, in this
paper, we promote modeless inference at the edge. The edge environment
introduces additional challenges related to low power consumption, limited
device memory, and volatile network environments.
  To address these challenges, we propose HawkVision, which provides
low-latency modeless serving of vision DNNs. HawkVision leverages a two-layer
edge-DC architecture that employs confidence scaling to reduce the number of
model options while meeting diverse accuracy requirements. It also supports
lossy inference under volatile network environments. Our experimental results
show that HawkVision outperforms current serving systems by up to 1.6X in P99
latency for providing modeless service. Our FPGA prototype demonstrates similar
performance at certain accuracy levels with up to a 3.34X reduction in power
consumption.
