  The emerging field of artificial intelligence of things (AIoT, AI+IoT) is
driven by the widespread use of intelligent infrastructures and the impressive
success of deep learning (DL). With the deployment of DL on various intelligent
infrastructures featuring rich sensors and weak DL computing capabilities, a
diverse range of AIoT applications has become possible. However, DL models are
notoriously resource-intensive. Existing research strives to realize
near-/realtime inference of AIoT live data and low-cost training using AIoT
datasets on resource-scare infrastructures. Accordingly, the accuracy and
responsiveness of DL models are bounded by resource availability. To this end,
the algorithm-system co-design that jointly optimizes the resource-friendly DL
models and model-adaptive system scheduling improves the runtime resource
availability and thus pushes the performance boundary set by the standalone
level. Unlike previous surveys on resource-friendly DL models or hand-crafted
DL compilers/frameworks with partially fine-tuned components, this survey aims
to provide a broader optimization space for more free resource-performance
tradeoffs. The cross-level optimization landscape involves various granularity,
including the DL model, computation graph, operator, memory schedule, and
hardware instructor in both on-device and distributed paradigms. Furthermore,
due to the dynamic nature of AIoT context, which includes heterogeneous
hardware, agnostic sensing data, varying user-specified performance demands,
and resource constraints, this survey explores the context-aware
inter-/intra-device controllers for automatic cross-level adaptation.
Additionally, we identify some potential directions for resource-efficient AIoT
systems. By consolidating problems and techniques scattered over diverse
levels, we aim to help readers understand their connections and stimulate
further discussions.
