  When facing changing environments in the real world, the lightweight model on
client devices suffers from severe performance drops under distribution shifts.
The main limitations of the existing device model lie in (1) unable to update
due to the computation limit of the device, (2) the limited generalization
ability of the lightweight model. Meanwhile, recent large models have shown
strong generalization capability on the cloud while they can not be deployed on
client devices due to poor computation constraints. To enable the device model
to deal with changing environments, we propose a new learning paradigm of
Cloud-Device Collaborative Continual Adaptation, which encourages collaboration
between cloud and device and improves the generalization of the device model.
Based on this paradigm, we further propose an Uncertainty-based Visual Prompt
Adapted (U-VPA) teacher-student model to transfer the generalization capability
of the large model on the cloud to the device model. Specifically, we first
design the Uncertainty Guided Sampling (UGS) to screen out challenging data
continuously and transmit the most out-of-distribution samples from the device
to the cloud. Then we propose a Visual Prompt Learning Strategy with
Uncertainty guided updating (VPLU) to specifically deal with the selected
samples with more distribution shifts. We transmit the visual prompts to the
device and concatenate them with the incoming data to pull the device testing
distribution closer to the cloud training distribution. We conduct extensive
experiments on two object detection datasets with continually changing
environments. Our proposed U-VPA teacher-student framework outperforms previous
state-of-the-art test time adaptation and device-cloud collaboration methods.
The code and datasets will be released.
