  Many large vision models have been deployed on the cloud for real-time
services. Meanwhile, fresh samples are continuously generated on the served
mobile device. How to leverage the device-side samples to improve the
cloud-side large model becomes a practical requirement, but falls into the
dilemma of no raw sample up-link and no large model down-link. Specifically,
the user may opt out of sharing raw samples with the cloud due to the concern
of privacy or communication overhead, while the size of some large vision
models far exceeds the mobile device's runtime capacity. In this work, we
propose a device-cloud collaborative controlled learning framework, called
DC-CCL, enabling a cloud-side large vision model that cannot be directly
deployed on the mobile device to still benefit from the device-side local
samples. In particular, DC-CCL vertically splits the base model into two
submodels, one large submodel for learning from the cloud-side samples and the
other small submodel for learning from the device-side samples and performing
device-cloud knowledge fusion. Nevertheless, on-device training of the small
submodel requires the output of the cloud-side large submodel to compute the
desired gradients. DC-CCL thus introduces a light-weight model to mimic the
large cloud-side submodel with knowledge distillation, which can be offloaded
to the mobile device to control its small submodel's optimization direction.
Given the decoupling nature of two submodels in collaborative learning, DC-CCL
also allows the cloud to take a pre-trained model and the mobile device to take
another model with a different backbone architecture.
