  Smart IoT-based systems often desire continuous execution of multiple
latency-sensitive Deep Learning (DL) applications. The edge servers serve as
the cornerstone of such IoT-based systems, however, their resource limitations
hamper the continuous execution of multiple (multi-tenant) DL applications. The
challenge is that, DL applications function based on bulky "neural network (NN)
models" that cannot be simultaneously maintained in the limited memory space of
the edge. Accordingly, the main contribution of this research is to overcome
the memory contention challenge, thereby, meeting the latency constraints of
the DL applications without compromising their inference accuracy. We propose
an efficient NN model management framework, called Edge-MultiAI, that ushers
the NN models of the DL applications into the edge memory such that the degree
of multi-tenancy and the number of warm-starts are maximized. Edge-MultiAI
leverages NN model compression techniques, such as model quantization, and
dynamically loads NN models for DL applications to stimulate multi-tenancy on
the edge server. We also devise a model management heuristic for Edge-MultiAI,
called iWS-BFE, that functions based on the Bayesian theory to predict the
inference requests for multi-tenant applications, and uses it to choose the
appropriate NN models for loading, hence, increasing the number of warm-start
inferences. We evaluate the efficacy and robustness of Edge-MultiAI under
various configurations. The results reveal that Edge-MultiAI can stimulate the
degree of multi-tenancy on the edge by at least 2X and increase the number of
warm-starts by around 60% without any major loss on the inference accuracy of
the applications.
