  Deep learning models are increasingly deployed to edge devices for real-time
applications. To ensure stable service quality across diverse edge
environments, it is highly desirable to generate tailored model architectures
for different conditions. However, conventional pre-deployment model generation
approaches are not satisfactory due to the difficulty of handling the diversity
of edge environments and the demand for edge information. In this paper, we
propose to adapt the model architecture after deployment in the target
environment, where the model quality can be precisely measured and private edge
data can be retained. To achieve efficient and effective edge model generation,
we introduce a pretraining-assisted on-cloud model elastification method and an
edge-friendly on-device architecture search method. Model elastification
generates a high-quality search space of model architectures with the guidance
of a developer-specified oracle model. Each subnet in the space is a valid
model with different environment affinity, and each device efficiently finds
and maintains the most suitable subnet based on a series of edge-tailored
optimizations. Extensive experiments on various edge devices demonstrate that
our approach is able to achieve significantly better accuracy-latency tradeoffs
(e.g. 46.74\% higher on average accuracy with a 60\% latency budget) than
strong baselines with minimal overhead (13 GPU hours in the cloud and 2 minutes
on the edge server).
