  Machine learning (ML) has entered the mobile era where an enormous number of
ML models are deployed on edge devices. However, running common ML models on
edge devices continuously may generate excessive heat from the computation,
forcing the device to "slow down" to prevent overheating, a phenomenon called
thermal throttling. This paper studies the impact of thermal throttling on
mobile phones: when it occurs, the CPU clock frequency is reduced, and the
model inference latency may increase dramatically. This unpleasant inconsistent
behavior has a substantial negative effect on user experience, but it has been
overlooked for a long time. To counter thermal throttling, we propose to
utilize dynamic networks with shared weights and dynamically shift between
large and small ML models seamlessly according to their thermal profile, i.e.,
shifting to a small model when the system is about to throttle. With the
proposed dynamic shifting, the application runs consistently without
experiencing CPU clock frequency degradation and latency increase. In addition,
we also study the resulting accuracy when dynamic shifting is deployed and show
that our approach provides a reasonable trade-off between model latency and
model accuracy.
