<p>Unfortunately, <code>gl_FragDepth</code> won't enable you to write outside of the rasterized triangle's screen-space bounds. All it enables is to modify the values written to the depth buffer within the triangle, replacing the triangle's true depth with a shader-calculated value.</p>

<p>What you're seeing in the screenshot in the Unity forum thread you linked to is two intersecting parallax-mapped surfaces, where the depth output enables occlusion between the two surfaces to be correctly resolved in the screen-space region they both cover. But if you look at the right side of the screenshot, you can see the straight-line edge of the bottom brick polygon.</p>

<p>However! You might still be able to accomplish the goal of getting correct silhouette edges by using <code>discard</code> in the fragment shader when the parallax ray gets outside the bounds of the original polygon. This would involve modifying the ray-march code to check if the march has crossed a triangle edge, and exit with <code>discard</code> if so, rather than continuing to march.</p>

<p>You'd have to find a way to get the information about triangle edges into the fragment shader, though, as this isn't provided by OpenGL. Maybe make your vertex/index buffers available via SSBOs; then you can use <code>gl_PrimitiveID</code> in the fragment shader to index into them and lookup the vertex data.</p>

<p>Unfortunately I don't know offhand of any implementation of something like this, although I'm sure someone out there must have tried it before.</p>

CC BY-SA 3.0
2018-04-13T22:25:33.187Z
Thanks. Didn't know about discard - though after thinking about the problem, what I had in mind was returning a flag indicating that alpha should be set to 0 for "out of bounds", which I guess is a clumsy version of discard. I think this only works if the "height map" is interpreted as a "depth map", i.e. we always trace from the eye to points below the real x-y plane of the texture ... which I guess is fine! Can't try it yet, laptop kaput and awaiting new one.
barneypitt
CC BY-SA 3.0
2018-04-14T12:27:53.637Z
To clarify, you don't trace from the eye to below the x-y plane of the surface, you still trace to the x-y plane. But when determining whether a point on surface intercepts the ray, all points are below the surface. Thus some rays won't intercept the surface at all. In these cases, we discard/"transparify" them, thus creating a proper silhouette at the edges.
barneypitt
CC BY-SA 3.0
2018-04-14T12:28:03.817Z
Problems will still arise at the texture boundary with an adjacent texture whose height map doesn't tile with the current height map - you would see straight through the model at certain angles.  Though thinking about it, this would even be the case with "true" geometry from hardware tessellation of a height map - the problem is inherent to inferring geometry from height maps - I'm sure it is surmountable with some thought (and some work in the vertex shader).
barneypitt
CC BY-SA 3.0
2018-04-10T04:10:41.130Z
2018-04-10T04:10:41.130Z
Nathan Reed
<p>New to writing shaders, please forgive any naivety in the question!</p>

<p>Parallax mapping in any of its ray-marchy variants (e.g. steep parallax, cone step mapping, POM) generates fake geometry on a material, generally nearly indistinguishable from real geometry and at a much higher resolution than is practical using GPU tessellation. However, the geometry created does not extend outside the initial bounds of the triangle being rendered (unlike with GPU tessellation) - so e.g. a parallax mapped sphere still has circular edges.</p>

<p>I'm want to try adapting a parallax shader (preferably OpenGL, any high quality variation of parallax mapping will do though I gather cone stepping is pretty much the state of the art) to write outside of the poly bounds using gl_FragDepth (and enabling conservative depth output).</p>

<p>This seems to be theoretically possible (see this Unity example <a href="https://forum.unity.com/threads/modifying-depth-when-parallax-mapping.364037/" rel="nofollow noreferrer">Modifying depth when parallax mapping | Unity Community</a>). However that post did not include any shader code and got no responses.</p>

<p>Is anyone aware of an existing implementation of this (with source available), or else could give me a good idea of how I should proceed with implementing it?</p>

<p>Many thanks.   </p>

<shader><raymarching>
Parallax/relief mapping with gl_FragDepth and conservative depth
barneypitt
# Parallax/relief mapping with gl_FragDepth and conservative depth

(Asked by: username_0 on 2018-03-23)

New to writing shaders, please forgive any naivety in the question!

Parallax mapping in any of its ray-marchy variants (e.g. steep parallax, cone step mapping, POM) generates fake geometry on a material, generally nearly indistinguishable from real geometry and at a much higher resolution than is practical using GPU tessellation. However, the geometry created does not extend outside the initial bounds of the triangle being rendered (unlike with GPU tessellation) - so e.g. a parallax mapped sphere still has circular edges.

I'm want to try adapting a parallax shader (preferably OpenGL, any high quality variation of parallax mapping will do though I gather cone stepping is pretty much the state of the art) to write outside of the poly bounds using gl\_FragDepth (and enabling conservative depth output).

This seems to be theoretically possible (see this Unity example [Modifying depth when parallax mapping | Unity Community](https://forum.unity.com/threads/modifying-depth-when-parallax-mapping.364037/)). However that post did not include any shader code and got no responses.

Is anyone aware of an existing implementation of this (with source available), or else could give me a good idea of how I should proceed with implementing it?

Many thanks.

**Tags:** shader, raymarching


---

## 1 answers

### Answer by username_1 on 2018-04-10. Score: 1

Unfortunately, `gl_FragDepth` won't enable you to write outside of the rasterized triangle's screen-space bounds. All it enables is to modify the values written to the depth buffer within the triangle, replacing the triangle's true depth with a shader-calculated value.

What you're seeing in the screenshot in the Unity forum thread you linked to is two intersecting parallax-mapped surfaces, where the depth output enables occlusion between the two surfaces to be correctly resolved in the screen-space region they both cover. But if you look at the right side of the screenshot, you can see the straight-line edge of the bottom brick polygon.

However! You might still be able to accomplish the goal of getting correct silhouette edges by using `discard` in the fragment shader when the parallax ray gets outside the bounds of the original polygon. This would involve modifying the ray-march code to check if the march has crossed a triangle edge, and exit with `discard` if so, rather than continuing to march.

You'd have to find a way to get the information about triangle edges into the fragment shader, though, as this isn't provided by OpenGL. Maybe make your vertex/index buffers available via SSBOs; then you can use `gl_PrimitiveID` in the fragment shader to index into them and lookup the vertex data.

Unfortunately I don't know offhand of any implementation of something like this, although I'm sure someone out there must have tried it before.

> **Comments:**
> **username_0**: Thanks. Didn't know about discard - though after thinking about the problem, what I had in mind was returning a flag indicating that alpha should be set to 0 for "out of bounds", which I guess is a clumsy version of discard. I think this only works if the "height map" is interpreted as a "depth map", i.e. we always trace from the eye to points below the real x-y plane of the texture ... which I guess is fine! Can't try it yet, laptop kaput and awaiting new one.  
> **username_0**: To clarify, you don't trace from the eye to below the x-y plane of the surface, you still trace to the x-y plane. But when determining whether a point on surface intercepts the ray, all points are below the surface. Thus some rays won't intercept the surface at all. In these cases, we discard/"transparify" them, thus creating a proper silhouette at the edges.  
> **username_0**: Problems will still arise at the texture boundary with an adjacent texture whose height map doesn't tile with the current height map - you would see straight through the model at certain angles.  Though thinking about it, this would even be the case with "true" geometry from hardware tessellation of a height map - the problem is inherent to inferring geometry from height maps - I'm sure it is surmountable with some thought (and some work in the vertex shader).  

---

