  In this paper, we study the problem of continuous 3D shape representations.
The majority of existing successful methods are coordinate-based implicit
neural representations. However, they are inefficient to render novel views or
recover explicit surface points. A few works start to formulate 3D shapes as
ray-based neural functions, but the learned structures are inferior due to the
lack of multi-view geometry consistency. To tackle these challenges, we propose
a new framework called RayDF. It consists of three major components: 1) the
simple ray-surface distance field, 2) the novel dual-ray visibility classifier,
and 3) a multi-view consistency optimization module to drive the learned
ray-surface distances to be multi-view geometry consistent. We extensively
evaluate our method on three public datasets, demonstrating remarkable
performance in 3D surface point reconstruction on both synthetic and
challenging real-world 3D scenes, clearly surpassing existing coordinate-based
and ray-based baselines. Most notably, our method achieves a 1000x faster speed
than coordinate-based methods to render an 800x800 depth image, showing the
superiority of our method for 3D shape representation. Our code and data are
available at https://github.com/vLAR-group/RayDF
