  Low-resource settings are well-established in natural language processing,
where many languages lack sufficient data for deep learning at scale. However,
low-resource problems are under-explored in computer vision. In this paper, we
address this gap and explore the challenges of low-resource image tasks with
vision foundation models. We first collect a benchmark of genuinely
low-resource image data, covering historic maps, circuit diagrams, and
mechanical drawings. These low-resource settings all share three challenges:
data scarcity, fine-grained differences, and the distribution shift from
natural images to the specialized domain of interest. While existing foundation
models have shown impressive generalizability, we find they cannot transfer
well to our low-resource tasks. To begin to tackle the challenges of
low-resource vision, we introduce one simple baseline per challenge.
Specifically, we i) enlarge the data space by generative models, ii) adopt the
best sub-kernels to encode local regions for fine-grained difference discovery
and iii) learn attention for specialized domains. Experiments on our three
low-resource tasks demonstrate our proposals already provide a better baseline
than transfer learning, data augmentation, and fine-grained methods. This
highlights the unique characteristics and challenges of low-resource vision for
foundation models that warrant further investigation. Project page:
https://xiaobai1217.github.io/Low-Resource-Vision/.
