  The majority of IoT devices like smartwatches, smart plugs, HVAC controllers,
etc., are powered by hardware with a constrained specification (low memory,
clock speed and processor) which is insufficient to accommodate and execute
large, high-quality models. On such resource-constrained devices, manufacturers
still manage to provide attractive functionalities (to boost sales) by
following the traditional approach of programming IoT devices/products to
collect and transmit data (image, audio, sensor readings, etc.) to their
cloud-based ML analytics platforms. For decades, this online approach has been
facing issues such as compromised data streams, non-real-time analytics due to
latency, bandwidth constraints, costly subscriptions, recent privacy issues
raised by users and the GDPR guidelines, etc. In this paper, to enable
ultra-fast and accurate AI-based offline analytics on resource-constrained IoT
devices, we present an end-to-end multi-component model optimization sequence
and open-source its implementation. Researchers and developers can use our
optimization sequence to optimize high memory, computation demanding models in
multiple aspects in order to produce small size, low latency, low-power
consuming models that can comfortably fit and execute on resource-constrained
hardware. The experimental results show that our optimization components can
produce models that are; (i) 12.06 x times compressed; (ii) 0.13% to 0.27% more
accurate; (iii) Orders of magnitude faster unit inference at 0.06 ms. Our
optimization sequence is generic and can be applied to any state-of-the-art
models trained for anomaly detection, predictive maintenance, robotics, voice
recognition, and machine vision.
