<p>It seems this question is not going to be answered, therefore I am here adding the things I have learnt recently. I've discussed this &quot;software SER&quot; problem with a senior engine dev and read some docs and code, and here is the current conclusion:</p>
<p>SER is not impossible for a pure software implementation, yet it would be hard to make it fast. There are a few techniques that can be used:</p>
<ul>
<li>Stream compaction. This is quite common and can be seen in various renderers. Based on this idea, if we take one step further:</li>
<li>We sort the ray based on some criteria. For example, I've seen examples that sort the ray based on the hit materials. Though, naive sorting might be inefficient since we might be working on the slow global memory and sorting is not so GPU friendly (however, we have thrust). Radix sort might be preferred. Also, I took a look at the path tracing shader of <a href="https://github.com/NVIDIAGameWorks/Falcor" rel="nofollow noreferrer"><code>Falcor</code></a>, and in its <code>SER</code> module (I think it is based on OptiX?), we should feed the coherence hint to its API. The hint contains some of the flag, for example: (1) Whether the ray will be terminated by Russian Roulette? (2) Is the ray hitting the emissive geometry? (3) Is the ray active? This 'flag-based' coherence hint can be efficiently sorted by radix sort. It seems the thread reordering with also consider the hit material and the spatial position. Yet I don't know how the latter one is incorporated.</li>
<li>The whitepaper of NVIDIA does not clearly stated the memory optimizations made in SER, yet I believe that if we are going for high efficiency, the rays (or related information like coherence hints) can not reside on global memory . <a href="https://chipsandcheese.com/2023/05/16/shader-execution-reordering-nvidia-tackles-divergence/" rel="nofollow noreferrer">Shader Execution Reordering: Nvidia Tackles Divergence</a> mentions the use of L1 and L2 cache for saving the register states of threads before re-ordering, which... might not be directly implemented by software if we are to do global level sorting but maybe, for tiled-based rendering (one block for one tile), this might be done per-block through shared memory (unsure).</li>
</ul>
<p>Basically, if the inactive rays are discarded and the active rays are compacted and sorted by the possible correlation of instructions, a straightforward software SER can be implemented, yet some of the ops (like sorting) might introduce more overhead than improvement without hardware support.</p>
<p>More thoughts and suggestions are surely welcomed here. This post might be updated once I try the techniques mentioned above.</p>

CC BY-SA 4.0
2024-06-17T03:01:40.173Z
2024-06-17T03:01:40.173Z
Enigmatisms
<p>I am writing a software path tracer from scratch in CUDA (for learning purposes, without resorting to any higher-level graphics API like OptiX), and it is well-known that path tracing can have extremely unbalanced workloads within a warp (or wavefront), so the GPGPU implementation for path tracing can be far from optimal, in terms of work grouping for SIMT. I am aware that NVIDIA provides Shader Execution Reordering (SER) that employs coherent hints to sort the threads and reload them to SMs for better warp coherence, and I am very curious about <strong>how this technique is implemented under the hood</strong>. Specifically, I am looking for insights that help:</p>
<ul>
<li>Implement a software thread grouping / reordering method on GPU, for ray tracing.</li>
<li>(Or) understand how to efficiently dispatch shader (particularly, ray tracing) workloads between threads on GPU.</li>
<li>Since this is basically reinventing the wheel for learning purposes, getting this done via specific graphics API won't help.</li>
</ul>
<p>Note that the whitebook of NVIDIA SER does not provide specifics about how things can be done with pure software solution (maybe SER is offered by their hardware), and there seems to be few helpful articles about it... So any insight helps. I did ask GPT4 about this question, and it told me to implement a TaskQueue on GPU, which seems... pretty odd and inefficient to me.</p>

<raytracing><pathtracing><cuda>
How is Shader Execution Reordering implemented under the hood?
Enigmatisms
# How is Shader Execution Reordering implemented under the hood?

(Asked by: username_0 on 2024-06-11)

I am writing a software path tracer from scratch in CUDA (for learning purposes, without resorting to any higher-level graphics API like OptiX), and it is well-known that path tracing can have extremely unbalanced workloads within a warp (or wavefront), so the GPGPU implementation for path tracing can be far from optimal, in terms of work grouping for SIMT. I am aware that NVIDIA provides Shader Execution Reordering (SER) that employs coherent hints to sort the threads and reload them to SMs for better warp coherence, and I am very curious about **how this technique is implemented under the hood**. Specifically, I am looking for insights that help:

* Implement a software thread grouping / reordering method on GPU, for ray tracing.
* (Or) understand how to efficiently dispatch shader (particularly, ray tracing) workloads between threads on GPU.
* Since this is basically reinventing the wheel for learning purposes, getting this done via specific graphics API won't help.

Note that the whitebook of NVIDIA SER does not provide specifics about how things can be done with pure software solution (maybe SER is offered by their hardware), and there seems to be few helpful articles about it... So any insight helps. I did ask GPT4 about this question, and it told me to implement a TaskQueue on GPU, which seems... pretty odd and inefficient to me.

**Tags:** raytracing, pathtracing, cuda


---

## 1 answers

### Answer by username_0 on 2024-06-17. Score: 1

It seems this question is not going to be answered, therefore I am here adding the things I have learnt recently. I've discussed this "software SER" problem with a senior engine dev and read some docs and code, and here is the current conclusion:

SER is not impossible for a pure software implementation, yet it would be hard to make it fast. There are a few techniques that can be used:

* Stream compaction. This is quite common and can be seen in various renderers. Based on this idea, if we take one step further:
* We sort the ray based on some criteria. For example, I've seen examples that sort the ray based on the hit materials. Though, naive sorting might be inefficient since we might be working on the slow global memory and sorting is not so GPU friendly (however, we have thrust). Radix sort might be preferred. Also, I took a look at the path tracing shader of [`Falcor`](https://github.com/NVIDIAGameWorks/Falcor), and in its `SER` module (I think it is based on OptiX?), we should feed the coherence hint to its API. The hint contains some of the flag, for example: (1) Whether the ray will be terminated by Russian Roulette? (2) Is the ray hitting the emissive geometry? (3) Is the ray active? This 'flag-based' coherence hint can be efficiently sorted by radix sort. It seems the thread reordering with also consider the hit material and the spatial position. Yet I don't know how the latter one is incorporated.
* The whitepaper of NVIDIA does not clearly stated the memory optimizations made in SER, yet I believe that if we are going for high efficiency, the rays (or related information like coherence hints) can not reside on global memory . [Shader Execution Reordering: Nvidia Tackles Divergence](https://chipsandcheese.com/2023/05/16/shader-execution-reordering-nvidia-tackles-divergence/) mentions the use of L1 and L2 cache for saving the register states of threads before re-ordering, which... might not be directly implemented by software if we are to do global level sorting but maybe, for tiled-based rendering (one block for one tile), this might be done per-block through shared memory (unsure).

Basically, if the inactive rays are discarded and the active rays are compacted and sorted by the possible correlation of instructions, a straightforward software SER can be implemented, yet some of the ops (like sorting) might introduce more overhead than improvement without hardware support.

More thoughts and suggestions are surely welcomed here. This post might be updated once I try the techniques mentioned above.


---

