  Differentiable renderers have been used successfully for unsupervised 3D
structure learning from 2D images because they can bridge the gap between 3D
and 2D. To optimize 3D shape parameters, current renderers rely on pixel-wise
losses between rendered images of 3D reconstructions and ground truth images
from corresponding viewpoints. Hence they require interpolation of the
recovered 3D structure at each pixel, visibility handling, and optionally
evaluating a shading model. In contrast, here we propose a Differentiable
Renderer Without Rendering (DRWR) that omits these steps. DRWR only relies on a
simple but effective loss that evaluates how well the projections of
reconstructed 3D point clouds cover the ground truth object silhouette.
Specifically, DRWR employs a smooth silhouette loss to pull the projection of
each individual 3D point inside the object silhouette, and a structure-aware
repulsion loss to push each pair of projections that fall inside the silhouette
far away from each other. Although we omit surface interpolation, visibility
handling, and shading, our results demonstrate that DRWR achieves
state-of-the-art accuracies under widely used benchmarks, outperforming
previous methods both qualitatively and quantitatively. In addition, our
training times are significantly lower due to the simplicity of DRWR.
