  Understanding geometric concepts, such as distance and shape, is essential
for understanding the real world and also for many vision tasks. To incorporate
such information into a visual representation of a scene, we propose learning
to represent the scene by sketching, inspired by human behavior. Our method,
coined Learning by Sketching (LBS), learns to convert an image into a set of
colored strokes that explicitly incorporate the geometric information of the
scene in a single inference step without requiring a sketch dataset. A sketch
is then generated from the strokes where CLIP-based perceptual loss maintains a
semantic similarity between the sketch and the image. We show theoretically
that sketching is equivariant with respect to arbitrary affine transformations
and thus provably preserves geometric information. Experimental results show
that LBS substantially improves the performance of object attribute
classification on the unlabeled CLEVR dataset, domain transfer between CLEVR
and STL-10 datasets, and for diverse downstream tasks, confirming that LBS
provides rich geometric information.
