<p>There is no universal solution here since there are many different small variations that can be applied to get a specific result.</p>
<p>One of the things that trips people up when computing ray's in the vertex shader is how a vector's length changes when it is interpolated. This results in vectors that have slightly different lengths arriving in the fragment shader. This is the same problem that happens with lighting vectors. With lighting the solution is to re-normalize the vector in the fragment shader. With ray casting the vector length needs to be recomputed to match the needs of the technique. This means we can effectively ignore ray length in the vertex shader. But the code must be careful to compute the proper length in the fragment shader.</p>
<p>One good technique is to compute the ray direction in the vertex shader and pass it to the fragment shader. The ray origin can then be computed in the fragment shader in a straight forward manner. Where ray direction is computed by finding the point of intersection with the camera space starting plane in 2D (not necessarily the projection near plane) and just plugging the distance from the camera to the plane in as the 'z' value. The vector length will be odd but its going to be changed with interpolation anyway, then transform it into the appropriate space for interpolation.</p>
<p>Ray origin can be computed in the fragment shader after rescaling (or computing a scaling factor) for the interpolated ray. A scaling factor can be computed with something along the lines of</p>
<p><code>scale = plane_distance * (1/camera_vector_length)</code></p>
<p>And then the Ray origin computed with:</p>
<p><code>ray_origin = interpolated_ray_direction * scale + transformed_camera_postion</code></p>
<p>It can be better to just pass a camera space 2D &quot;ray&quot; and another ray transformed into the same space that the ray will be marched/cast in so that there is no need to do any transformations of the ray in the fragment shader and is the reason I wrote &quot;camera_vector_length&quot; above. The 2D ray is really just interpolating a point across the start plane then the plane distance is plugged in as the 'z' value in the fragment shader. (saving some interpolation hardware) Also, the <code>transformed_camera_position</code> above, is the camera position transformed into whatever space the ray marching will be done in.</p>
<p>Another nice technique is to compute the ray end point, then compute a time <code>t</code> that is used to interpolate between the start and end points. Where <code>t</code> is a step size that can be set in the shader, or passed in as a parameter, computed on the fly, etc. The interpolated point is then used to sample the data. This drops ray direction out of the marching loops altogether.</p>
<p>Yet another reason to use <code>t</code> is that a random offset can be <em>easily</em> added to it which helps &quot;jitter&quot; the sampling points breaking up regular artifacts that crop up often with ray casting. The goal hear is to move the rays slightly from frame to frame rather then within a single frame.</p>
<p>Using these approach's results in shaders that look considerably different from the standard &quot;Rd = hitpos - camerapos&quot; but are more efficient and produce better results.</p>
<hr />
<pre><code>// out cs_ray == camera space ray
// out t_ray == transformed ray created using transform matrix
//    such as the camera to world matrix, or camera to shadow space matrix
void ComputeRayFromClipSpace( in float2 vp, out float2 cs_ray, out float3 t_ray ) 
{ 
  // Inputs to shader
  float pd; // distance to camera space plane, ie Ray Origin plane
  float s;  // aspect ratio
  float g;  // focal distance

  // Compute point on camera space plane
  float3 hitpoint = float3( vp*pd*s/g, vp*pd*1.0/g, pd);

  cs_ray = hitpoint.xy;
  t_ray  = hitpoint * transform_matrix;
}
</code></pre>
<p>This is setup for a &quot;full screen&quot; effect.  The inputs would usually be generated from a single clip space triangle that covers the entire vieport. <a href="https://wallisc.github.io/rendering/2021/04/18/Fullscreen-Pass.html" rel="nofollow noreferrer">More info here</a>. It is also using info from a perspective frustum but similar info could be generated for a orthographic frustum. The main point here is to compute the points on the plane that intersect with the frustum using whatever method that is appropriate.</p>
<p>This function is leaving some performance on the table for clarity, the constants and transform computations.</p>
<p>Then in the fragment shader:</p>
<pre><code>float3 camera_ray = float3( cameraRayIn, pd ); // pd == plane distance
float length = sqrt( dot( camera_ray, camera_ray));
float scale = pd * (1.0/length);
float3 scaled_ray = t_ray_in * scale;
</code></pre>
<p>The scaled_ray can be used to compute the start point <code>scaled_ray+camera_pos</code> where camera_pos is the position of the camera in whatever space we are working in. The ray can then be rescaled (such as normalized or scaled to the required step distance) or used to compute a ray end point.</p>
<p>The best sources I know of for this technique are talks and books describing ray casting effects used in games like fged2.</p>

CC BY-SA 4.0
2023-08-27T16:26:19.837Z
Thanks for a detailed answer, I'll try to wrap my head around it and try the approaches you explained. Could you possibly provide some links to the implementations or some other resources about the two techniques you mentioned above? I noticed that a devil almost always is in the details. If it isn't too much to ask, could you maybe paste some example hlsl/glsl implementation of this in here?
Tooster
CC BY-SA 4.0
2023-08-27T16:28:49.927Z
Meanwhile, I have written a visualization in the graphing calculator to check how vectors calculated in vertex shader get interpolated when passed to fragment shader, and like I guessed (I may have done it wrong before, but I don't know anymore), vectors which are somehow changed in vertex shader (in my case normalized) get quite messed up in the fragment shader. Here is a quick demo, green is correct interpolated ray, red is wrong ray that resulted from interpolating normalized vectors https://www.geogebra.org/calculator/fppufpcf
(Under assumption that I correctly understand interpolation)
Tooster
CC BY-SA 4.0
2023-08-28T12:50:46.987Z
I added some code and some more description.
pmw1234
CC BY-SA 4.0
2023-08-29T09:46:06.523Z
Thank a lot, I'm gonna mark your answer as accepted when I get to test it and I get it to work  for now it must wait a bit for a bit of my spare time :P.
Tooster
CC BY-SA 4.0
2023-08-27T15:58:36.500Z
2023-08-28T12:48:37.243Z
2023-08-28T12:48:37.243Z
pmw1234
<p>I have been reading and watching many raymarching tutorials, but almost all of them are only working in the perspective projection. Almost all assume, that to calculate the ray direction, you just have to calculate the <code>rd = hitpos - cameraPos</code>, where hitpos is vertex position. This approach doesn't work when we change the projection to orthographic though.</p>
<p>Because there are a lot of hidden details in calculating and correctly interpolating the rays and positions, how to correctly calculate camera ray direction and ray origin like on the picture below (without any external scripts)? <code>Rd</code> is ray direction, <code>Ro</code> is ray origin, <code>near</code> and <code>far</code> are camera clip planes and <code>Hit</code> will be fragment's position.</p>
<p>I am mainly interested in HLSL implementation, specifically in Unity, but the general approach would be welcome as well.</p>
<p><a href="https://i.sstatic.net/FfSIH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FfSIH.png" alt="raymarching" /></a></p>
<p>The technique isn't trivial, I had to already deal with:</p>
<ul>
<li>it has to account for both orthographic and perspective projection at the same time</li>
<li>interpolating directions in vertex shader can lead to weird interpolation issues later in fragment (looking similarly to issues with perspective-correct texture mapping)</li>
<li>the non-trivial thing about how interpolators work for data passed from vertex to fragment</li>
<li>accounting for depth buffer (and how it's not linear)</li>
<li>using only data already available in the shader routines without passing things like FOV, camera position and similar to shader explicitly</li>
<li>working in many different coordinate systems (local, world, view, clip)</li>
<li>if a depth buffer is reversed or not</li>
</ul>
<p>And I still don't quite understand the correct way of accounting all of those things and calculating things like worldspace pos from depth.</p>
<p>I'm gonna mention, that I have already achieved desired effect, but in the fragment shader (origin on near plane or hit position, depending on control, direction universally calculated for perspective and ortho). It is overcomplicated and most probably very inefficient. It uses clip-space to generate a ray: it takes the fragment coordinates in clip-space, calculates direction vector from projecting this fragment to z=0 and z=1, then calculates an inverse projection matrix to bring this vector back to camera space.</p>
<hr />
<p><em>EDIT:</em></p>
<p>Since I asked this question I have implemented universal raymarching technique for perspective and ortho. I implemented is as a part of my open-source program, and the result you can find here:
<a href="https://github.com/T3sT3ro/SdfManipulator/blob/2cd6db0f79b29e7644ca7dec601fea5969a0e914/Packages/me.tooster.sdf/Editor/Resources/Includes/camera.hlsl#L36" rel="nofollow noreferrer">https://github.com/T3sT3ro/SdfManipulator/blob/2cd6db0f79b29e7644ca7dec601fea5969a0e914/Packages/me.tooster.sdf/Editor/Resources/Includes/camera.hlsl#L36</a></p>
<p>In a TL;DR fashion:</p>
<ul>
<li>use a control variable desscribing the type of projection (this can be extracted from a projection matrix parameters or passed as uniform). Fortunately unity has this out of the box.</li>
<li>depending on this control variable pick construct appropriate direction vector: in ortho tit's just the camera forward and and in perspective it's made from a triangle constructed from a camera forward vector and a vector lying on the near plane</li>
<li>if you calculate in vertex shader to automatically interpolate between dirs, take special care when interpolating (don't normalize the vectors in vertex shader) and when using interpolators (disable perspective correction, for example in HLSL use <code>noperspective</code> on a ray direction data passed from vertex to fragment)</li>
<li>This is probably the simplest and most performant way to implement it. Any approaches trying to use inverse matrices are a lot messier and slower + have edgecases.</li>
</ul>

<shader><hlsl><raymarching><unity>
How to calculate ray-origin and ray-direction in vertex shader working universally for any projection and using interpolators for performance
Tooster
# How to calculate ray-origin and ray-direction in vertex shader working universally for any projection and using interpolators for performance

(Asked by: username_0 on 2023-08-27)

I have been reading and watching many raymarching tutorials, but almost all of them are only working in the perspective projection. Almost all assume, that to calculate the ray direction, you just have to calculate the `rd = hitpos - cameraPos`, where hitpos is vertex position. This approach doesn't work when we change the projection to orthographic though.

Because there are a lot of hidden details in calculating and correctly interpolating the rays and positions, how to correctly calculate camera ray direction and ray origin like on the picture below (without any external scripts)? `Rd` is ray direction, `Ro` is ray origin, `near` and `far` are camera clip planes and `Hit` will be fragment's position.

I am mainly interested in HLSL implementation, specifically in Unity, but the general approach would be welcome as well.

[![raymarching](https://i.sstatic.net/FfSIH.png)](https://i.sstatic.net/FfSIH.png)

The technique isn't trivial, I had to already deal with:

* it has to account for both orthographic and perspective projection at the same time
* interpolating directions in vertex shader can lead to weird interpolation issues later in fragment (looking similarly to issues with perspective-correct texture mapping)
* the non-trivial thing about how interpolators work for data passed from vertex to fragment
* accounting for depth buffer (and how it's not linear)
* using only data already available in the shader routines without passing things like FOV, camera position and similar to shader explicitly
* working in many different coordinate systems (local, world, view, clip)
* if a depth buffer is reversed or not

And I still don't quite understand the correct way of accounting all of those things and calculating things like worldspace pos from depth.

I'm gonna mention, that I have already achieved desired effect, but in the fragment shader (origin on near plane or hit position, depending on control, direction universally calculated for perspective and ortho). It is overcomplicated and most probably very inefficient. It uses clip-space to generate a ray: it takes the fragment coordinates in clip-space, calculates direction vector from projecting this fragment to z=0 and z=1, then calculates an inverse projection matrix to bring this vector back to camera space.

---

*EDIT:*

Since I asked this question I have implemented universal raymarching technique for perspective and ortho. I implemented is as a part of my open-source program, and the result you can find here:
<https://github.com/T3sT3ro/SdfManipulator/blob/2cd6db0f79b29e7644ca7dec601fea5969a0e914/Packages/me.username_0.sdf/Editor/Resources/Includes/camera.hlsl#L36>

In a TL;DR fashion:

* use a control variable desscribing the type of projection (this can be extracted from a projection matrix parameters or passed as uniform). Fortunately unity has this out of the box.
* depending on this control variable pick construct appropriate direction vector: in ortho tit's just the camera forward and and in perspective it's made from a triangle constructed from a camera forward vector and a vector lying on the near plane
* if you calculate in vertex shader to automatically interpolate between dirs, take special care when interpolating (don't normalize the vectors in vertex shader) and when using interpolators (disable perspective correction, for example in HLSL use `noperspective` on a ray direction data passed from vertex to fragment)
* This is probably the simplest and most performant way to implement it. Any approaches trying to use inverse matrices are a lot messier and slower + have edgecases.

**Tags:** shader, hlsl, raymarching, unity


---

## 1 answers

### Answer by username_1 on 2023-08-27. Score: 2 | Accepted answer

There is no universal solution here since there are many different small variations that can be applied to get a specific result.

One of the things that trips people up when computing ray's in the vertex shader is how a vector's length changes when it is interpolated. This results in vectors that have slightly different lengths arriving in the fragment shader. This is the same problem that happens with lighting vectors. With lighting the solution is to re-normalize the vector in the fragment shader. With ray casting the vector length needs to be recomputed to match the needs of the technique. This means we can effectively ignore ray length in the vertex shader. But the code must be careful to compute the proper length in the fragment shader.

One good technique is to compute the ray direction in the vertex shader and pass it to the fragment shader. The ray origin can then be computed in the fragment shader in a straight forward manner. Where ray direction is computed by finding the point of intersection with the camera space starting plane in 2D (not necessarily the projection near plane) and just plugging the distance from the camera to the plane in as the 'z' value. The vector length will be odd but its going to be changed with interpolation anyway, then transform it into the appropriate space for interpolation.

Ray origin can be computed in the fragment shader after rescaling (or computing a scaling factor) for the interpolated ray. A scaling factor can be computed with something along the lines of

`scale = plane_distance * (1/camera_vector_length)`

And then the Ray origin computed with:

`ray_origin = interpolated_ray_direction * scale + transformed_camera_postion`

It can be better to just pass a camera space 2D "ray" and another ray transformed into the same space that the ray will be marched/cast in so that there is no need to do any transformations of the ray in the fragment shader and is the reason I wrote "camera\_vector\_length" above. The 2D ray is really just interpolating a point across the start plane then the plane distance is plugged in as the 'z' value in the fragment shader. (saving some interpolation hardware) Also, the `transformed_camera_position` above, is the camera position transformed into whatever space the ray marching will be done in.

Another nice technique is to compute the ray end point, then compute a time `t` that is used to interpolate between the start and end points. Where `t` is a step size that can be set in the shader, or passed in as a parameter, computed on the fly, etc. The interpolated point is then used to sample the data. This drops ray direction out of the marching loops altogether.

Yet another reason to use `t` is that a random offset can be *easily* added to it which helps "jitter" the sampling points breaking up regular artifacts that crop up often with ray casting. The goal hear is to move the rays slightly from frame to frame rather then within a single frame.

Using these approach's results in shaders that look considerably different from the standard "Rd = hitpos - camerapos" but are more efficient and produce better results.

---

```
// out cs_ray == camera space ray
// out t_ray == transformed ray created using transform matrix
//    such as the camera to world matrix, or camera to shadow space matrix
void ComputeRayFromClipSpace( in float2 vp, out float2 cs_ray, out float3 t_ray ) 
{ 
  // Inputs to shader
  float pd; // distance to camera space plane, ie Ray Origin plane
  float s;  // aspect ratio
  float g;  // focal distance

  // Compute point on camera space plane
  float3 hitpoint = float3( vp*pd*s/g, vp*pd*1.0/g, pd);

  cs_ray = hitpoint.xy;
  t_ray  = hitpoint * transform_matrix;
}

```

This is setup for a "full screen" effect. The inputs would usually be generated from a single clip space triangle that covers the entire vieport. [More info here](https://wallisc.github.io/rendering/2021/04/18/Fullscreen-Pass.html). It is also using info from a perspective frustum but similar info could be generated for a orthographic frustum. The main point here is to compute the points on the plane that intersect with the frustum using whatever method that is appropriate.

This function is leaving some performance on the table for clarity, the constants and transform computations.

Then in the fragment shader:

```
float3 camera_ray = float3( cameraRayIn, pd ); // pd == plane distance
float length = sqrt( dot( camera_ray, camera_ray));
float scale = pd * (1.0/length);
float3 scaled_ray = t_ray_in * scale;

```

The scaled\_ray can be used to compute the start point `scaled_ray+camera_pos` where camera\_pos is the position of the camera in whatever space we are working in. The ray can then be rescaled (such as normalized or scaled to the required step distance) or used to compute a ray end point.

The best sources I know of for this technique are talks and books describing ray casting effects used in games like fged2.

> **Comments:**
> **username_0**: Thanks for a detailed answer, I'll try to wrap my head around it and try the approaches you explained. Could you possibly provide some links to the implementations or some other resources about the two techniques you mentioned above? I noticed that a devil almost always is in the details. If it isn't too much to ask, could you maybe paste some example hlsl/glsl implementation of this in here?  
> **username_0**: Meanwhile, I have written a visualization in the graphing calculator to check how vectors calculated in vertex shader get interpolated when passed to fragment shader, and like I guessed (I may have done it wrong before, but I don't know anymore), vectors which are somehow changed in vertex shader (in my case normalized) get quite messed up in the fragment shader. Here is a quick demo, green is correct interpolated ray, red is wrong ray that resulted from interpolating normalized vectors https://www.geogebra.org/calculator/fppufpcf (Under assumption that I correctly understand interpolation)  
> **username_1**: I added some code and some more description.  
> **username_0**: Thank a lot, I'm gonna mark your answer as accepted when I get to test it and I get it to work  for now it must wait a bit for a bit of my spare time :P.  

---

