<p>There are a few confusing statements in your question. However, I'll still assume you're largely aware of how the stencil buffer and the stencil test actually work. I'll also talk about OpenGL, based on your tags, but the general principles should apply to any similar API.</p>
<p>First of all, you say</p>
<blockquote>
<p>...stencil buffer can only be filled in by some hardcoded rules during geometry drawing (like depth buffer)...</p>
</blockquote>
<p>But for the depth buffer that isn't actually true. You can very well decide what depth value you actually want to write by just writing anything to <code>gl_FragDepth</code> in your shader.</p>
<p>Now you're right that in unextended GL you can't just write arbitrary values to the stencil buffer like this. However, <strong>if your platform supports the <a href="https://registry.khronos.org/OpenGL/extensions/ARB/ARB_shader_stencil_export.txt" rel="nofollow noreferrer">GL_ARB_shader_stencil_export</a> extension, you can do exactly that by writing to <code>gl_FragStencilRefARB</code></strong>. Or more precisely, you write the reference value that is used in the stencil test. So if you have a stencil op of <code>GL_REPLACE</code>, you can basically just write the value per fragment.</p>
<p>However, even if that extension isn't present, you can achieve similar things for simple scenarios by merely <strong>deciding which fragments to write at all</strong>. You can just globally specificy whichever stencil reference value you want to write for your fragments and then in the fragment shader simply <strong><code>discard</code> any fragments for which you don't want to write anything</strong>. Of course this doesn't let you arbitrarily choose the value you want to write per fragment, but when using stencil testing you normally don't really need any such kind of fine-grained control over its bits (afterall, it's usually got just 256 different values anyway).</p>
<p>As to your other question:</p>
<blockquote>
<p>...is it possible to use stencil buffer separate of depth buffer in this case, don't use depth buffer at all?</p>
</blockquote>
<p>The stencil buffer really doesn't have anything to do with the depth buffer (at least functionally). Yes, the stencil op lets you choose different actions depending if the fragment passed the depth test or not. But <strong>noone requires you to use the depth test at all. If you don't need depth testing, just don't do it.</strong> Or if you don't need stencil testing and the depth buffer already works for you, then use that.</p>
<p>There is a bit of a practical consideration that usually the actual <strong>framebuffer is laid out so that the depth and stencil buffer share their storage</strong> in a combined depth-stencil buffer, because for the common case of a 24-bit depth buffer and an 8-bit stencil buffer this goes well together. It's what the window system usually does for you and what you should strive for when binding storage to framebuffer objects, too. However, that is merely the <strong>internal data layout and doesn't require you to always use depth and stencil testing together</strong>. It does, however, mean, that sparing away the stencil buffer usually doesn't save you any actual memory (at least in case of a common 24-bit depth format).</p>
<hr />
<p>As a side note, though, from your question I have a bit of hunch that you might not even need the stencil buffer to begin with. Maybe you weren't aware that you can just <code>discard</code> fragments in the fragment shader or decide yourself which depth value you want to write and maybe your possible multi-pass algorithm (since that's usually how stencil testing is used) can be done in a single pass or just differently altogether. If you'd need to write compilated stencil values from your shader, it could be a sign that the stencil buffer might not actually be the tool you're looking for.</p>

CC BY-SA 4.0
2023-10-06T02:06:15.343Z
Thank you so much for such a thoughtful explanation! Are you saying that using `discard` in fragment shader could achieve the same performance as stenciling fragments in rasterizer? I've read somewhere that usage of `discard` is discouraged specifically because of its poor performance. Something along the line of "use discard only for testing/debug purposes".
simd
CC BY-SA 4.0
2023-10-06T19:35:02.653Z
I honestly don't know about its performance implications. However, it's a common feature and if you need to discard fragments, then you need to discard fragments. There's hardly a way around it. Of course if you don't need it, then don't use it. It does come with some penalties, like e.g. disabling early depth testing. *If* you can do a simple stencil test instead, then maybe that's preferable, but your problem seems to be that your stencil scenario isn't simple to begin with anyway. But actual performance measurements might tell you more.
Christian Rau
CC BY-SA 4.0
2023-10-05T11:51:36.400Z
2023-10-05T11:51:36.400Z
Christian Rau
<p>I've never worked with stencil buffers, but I'm researching if it is the right tool for my task.</p>
<p>Basically I don't understand yet if stencil buffer can only be filled in by some hardcoded rules during geometry drawing (like depth buffer), or is possible to fill it in completely programmatically with the help of full-screen fragment shader, which will just draw black-and-white shape based on some 2D SDF? Also is it possible to use stencil buffer separate of depth buffer in this case, don't use depth buffer at all?</p>

<opengl><fragment-shader><metal><stencil-buffer><stencil-test>
Is it possible to fill in Stencil Buffer programmatically via fragment shader?
simd
# Is it possible to fill in Stencil Buffer programmatically via fragment shader?

(Asked by: username_0 on 2023-10-04)

I've never worked with stencil buffers, but I'm researching if it is the right tool for my task.

Basically I don't understand yet if stencil buffer can only be filled in by some hardcoded rules during geometry drawing (like depth buffer), or is possible to fill it in completely programmatically with the help of full-screen fragment shader, which will just draw black-and-white shape based on some 2D SDF? Also is it possible to use stencil buffer separate of depth buffer in this case, don't use depth buffer at all?

**Tags:** opengl, fragment-shader, metal, stencil-buffer, stencil-test


---

## 1 answers

### Answer by username_1 on 2023-10-05. Score: 3 | Accepted answer

There are a few confusing statements in your question. However, I'll still assume you're largely aware of how the stencil buffer and the stencil test actually work. I'll also talk about OpenGL, based on your tags, but the general principles should apply to any similar API.

First of all, you say

> ...stencil buffer can only be filled in by some hardcoded rules during geometry drawing (like depth buffer)...

But for the depth buffer that isn't actually true. You can very well decide what depth value you actually want to write by just writing anything to `gl_FragDepth` in your shader.

Now you're right that in unextended GL you can't just write arbitrary values to the stencil buffer like this. However, **if your platform supports the [GL\_ARB\_shader\_stencil\_export](https://registry.khronos.org/OpenGL/extensions/ARB/ARB_shader_stencil_export.txt) extension, you can do exactly that by writing to `gl_FragStencilRefARB`**. Or more precisely, you write the reference value that is used in the stencil test. So if you have a stencil op of `GL_REPLACE`, you can basically just write the value per fragment.

However, even if that extension isn't present, you can achieve similar things for simple scenarios by merely **deciding which fragments to write at all**. You can just globally specificy whichever stencil reference value you want to write for your fragments and then in the fragment shader simply **`discard` any fragments for which you don't want to write anything**. Of course this doesn't let you arbitrarily choose the value you want to write per fragment, but when using stencil testing you normally don't really need any such kind of fine-grained control over its bits (afterall, it's usually got just 256 different values anyway).

As to your other question:

> ...is it possible to use stencil buffer separate of depth buffer in this case, don't use depth buffer at all?

The stencil buffer really doesn't have anything to do with the depth buffer (at least functionally). Yes, the stencil op lets you choose different actions depending if the fragment passed the depth test or not. But **noone requires you to use the depth test at all. If you don't need depth testing, just don't do it.** Or if you don't need stencil testing and the depth buffer already works for you, then use that.

There is a bit of a practical consideration that usually the actual **framebuffer is laid out so that the depth and stencil buffer share their storage** in a combined depth-stencil buffer, because for the common case of a 24-bit depth buffer and an 8-bit stencil buffer this goes well together. It's what the window system usually does for you and what you should strive for when binding storage to framebuffer objects, too. However, that is merely the **internal data layout and doesn't require you to always use depth and stencil testing together**. It does, however, mean, that sparing away the stencil buffer usually doesn't save you any actual memory (at least in case of a common 24-bit depth format).

---

As a side note, though, from your question I have a bit of hunch that you might not even need the stencil buffer to begin with. Maybe you weren't aware that you can just `discard` fragments in the fragment shader or decide yourself which depth value you want to write and maybe your possible multi-pass algorithm (since that's usually how stencil testing is used) can be done in a single pass or just differently altogether. If you'd need to write compilated stencil values from your shader, it could be a sign that the stencil buffer might not actually be the tool you're looking for.

> **Comments:**
> **username_0**: Thank you so much for such a thoughtful explanation! Are you saying that using `discard` in fragment shader could achieve the same performance as stenciling fragments in rasterizer? I've read somewhere that usage of `discard` is discouraged specifically because of its poor performance. Something along the line of "use discard only for testing/debug purposes".  
> **username_1**: I honestly don't know about its performance implications. However, it's a common feature and if you need to discard fragments, then you need to discard fragments. There's hardly a way around it. Of course if you don't need it, then don't use it. It does come with some penalties, like e.g. disabling early depth testing. *If* you can do a simple stencil test instead, then maybe that's preferable, but your problem seems to be that your stencil scenario isn't simple to begin with anyway. But actual performance measurements might tell you more.  

---

