  Recent vision architectures and self-supervised training methods enable
vision models that are extremely accurate and general, but come with massive
parameter and computational costs. In practical settings, such as camera traps,
users have limited resources, and may fine-tune a pretrained model on (often
limited) data from a small set of specific categories of interest. These users
may wish to make use of modern, highly-accurate models, but are often
computationally constrained. To address this, we ask: can we quickly compress
large generalist models into accurate and efficient specialists? For this, we
propose a simple and versatile technique called Few-Shot Task-Aware Compression
(TACO). Given a large vision model that is pretrained to be accurate on a broad
task, such as classification over ImageNet-22K, TACO produces a smaller model
that is accurate on specialized tasks, such as classification across vehicle
types or animal species. Crucially, TACO works in few-shot fashion, i.e. only a
few task-specific samples are used, and the procedure has low computational
overheads. We validate TACO on highly-accurate ResNet, ViT/DeiT, and ConvNeXt
models, originally trained on ImageNet, LAION, or iNaturalist, which we
specialize and compress to a diverse set of "downstream" subtasks. TACO can
reduce the number of non-zero parameters in existing models by up to 20x
relative to the original models, leading to inference speedups of up to
3$\times$, while remaining accuracy-competitive with the uncompressed models on
the specialized tasks.
