<p>The context of this question is correct dithering of color with gamma correction. In dithering we add (shaped) noise <span class="math-container">$n$</span> to a signal <span class="math-container">$s \in [0, 1]$</span> based on a random variable <span class="math-container">$r \in [0, 1]$</span></p>

<p><span class="math-container">$$s' = s + n(s, r)$$</span></p>

<p>such that when we quantize the signal to <span class="math-container">$m$</span> steps the rounding will end up distorting the signal less.</p>

<p>I already have a function that works in a linear world from <a href="https://computergraphics.stackexchange.com/questions/5904/whats-a-proper-way-to-clamp-dither-noise/8777#8777">here</a>, where <span class="math-container">$n$</span> is a symmetric triangular distribution on <span class="math-container">$[-1, 1]$</span> (with the caveat that for <span class="math-container">$s$</span> close to <span class="math-container">$0$</span> or <span class="math-container">$1$</span> we switch to a uniform distribution on <span class="math-container">$[-\frac{1}{2}, \frac{1}{2}]$</span>), as it has mean <span class="math-container">$0$</span> and appropriately smoothes out the rounding (demo that incorrectly ignores gamma: <a href="https://www.shadertoy.com/view/Wts3zH" rel="nofollow noreferrer">https://www.shadertoy.com/view/Wts3zH</a>). This function written out is (matching top to bottom in priority for the cases):</p>

<p><span class="math-container">$$n(s, r) \cdot (m - 1) = \begin{cases}
r - \frac{1}{2}&amp;s \leq \frac{1}{2(m-1)} \vee s \geq 1-\frac{1}{2(m-1)}\\
\sqrt{2r} - 1&amp;r &lt; \frac{1}{2}\\
1 - \sqrt{2 - 2r}&amp;r \geq \frac{1}{2}\\
\end{cases}$$</span></p>

<p>However in sRGB files we have to deal with gamma. Essentially we have an invertable transform <span class="math-container">$t$</span> (often <span class="math-container">$t(x) = x^{2.4}$</span>) that is applied before/after quantization to allocate more steps to lower intensities.</p>

<p>This means that the average color when viewing is no longer (ignoring quantization for now):</p>

<p><span class="math-container">$$\int_0^1\left(s + n(s, r)\right)\text{d}r = s$$</span></p>

<p>But is instead:</p>

<p><span class="math-container">$$\int_0^1 t^{-1}\left(s + n(s, r)\right)\text{d}r &gt; s$$</span></p>

<p>How can we change our noise function <span class="math-container">$n$</span> so that the average color is again <span class="math-container">$s$</span> after dithering, while maintaining our properties from the triangular distribution as much as possible?</p>

<p>For the record, the main property the triangular distribution gives is that the variance after rounding becomes constant:</p>

<p><span class="math-container">$$\int_0^1\left(\left[(s + n(s, r))(m-1)\right]/(m-1) - s\right)^2\text{d}r = c$$</span></p>

<p>Where <span class="math-container">$[x]$</span> denotes <span class="math-container">$x$</span> rounded to the closest integer. Note that this only holds in the triangular part of the domain of <span class="math-container">$n$</span>, so when <span class="math-container">$\frac{1}{2(m-1)} \leq s \leq 1-\frac{1}{2(m-1)}$</span>.</p>

<p>If you want to give a shot, here is a demo that can show you the error from dithering, which has gamma correction: <a href="https://www.shadertoy.com/view/3tf3Dn" rel="nofollow noreferrer">https://www.shadertoy.com/view/3tf3Dn</a>.</p>

CC BY-SA 4.0
2019-05-01T09:13:32.170Z
I think now I understand your question. I'm not familiar with the language you implemented the function you mentioned, could you maybe write down $n(s,r)$ explicitly in "math notation"? :) (btw: I assume $r$ is uniform on $[0,1]$, right?)
flawr
CC BY-SA 4.0
2019-05-01T09:49:27.887Z
@flawr It's only `dither_quantize` you need to look at. There `c` is $s$, `rng` is $r$, `depth` is $m$, `ci` is $s\cdot (m-1)$ and `d` is $n(s, r)\cdot (m-1)$. `a ? b : c` means `b` if `a` else `c`, and `clamp(x, lo, hi)` is `min(max(x, lo), hi)`. Finally `uint(x)` is basically $\lfloor x \rfloor$.
orlp
CC BY-SA 4.0
2019-05-01T09:54:55.200Z
@flawr One thing that isn't particularly clean here is that we must not forget $n(s, r)$ also depends on $m$, as fewer quantization steps means the noise must increase in size. It is $n(s, r) \cdot (m-1)$ that doesn't change as $m$ does.
orlp
CC BY-SA 4.0
2019-05-01T10:26:21.840Z
@flawr There is another way you can view it, which I posted here: https://math.stackexchange.com/questions/3200249/maintaining-probability-distribution-under-transform. I convolved the above noise function with a unit square to find the probability that a signal $s$ gets quantized to $[s]$ after adding noise. Then you can view the whole quantizing operation as summing a couple random variables (e.g. one for the chance at $[s] -1$, one for $[s]$ and one for $[s] + 1$).
orlp
<noise><gamma>
Noise shaping for dithering with gamma?
orlp
# Noise shaping for dithering with gamma?

(Asked by: username_0 on 2019-04-23)

The context of this question is correct dithering of color with gamma correction. In dithering we add (shaped) noise $n$ to a signal $s \in [0, 1]$ based on a random variable $r \in [0, 1]$

$$s' = s + n(s, r)$$

such that when we quantize the signal to $m$ steps the rounding will end up distorting the signal less.

I already have a function that works in a linear world from [here](https://computergraphics.stackexchange.com/questions/5904/whats-a-proper-way-to-clamp-dither-noise/8777#8777), where $n$ is a symmetric triangular distribution on $[-1, 1]$ (with the caveat that for $s$ close to $0$ or $1$ we switch to a uniform distribution on $[-\frac{1}{2}, \frac{1}{2}]$), as it has mean $0$ and appropriately smoothes out the rounding (demo that incorrectly ignores gamma: <https://www.shadertoy.com/view/Wts3zH>). This function written out is (matching top to bottom in priority for the cases):

$$n(s, r) \cdot (m - 1) = \begin{cases}
r - \frac{1}{2}&s \leq \frac{1}{2(m-1)} \vee s \geq 1-\frac{1}{2(m-1)}\\
\sqrt{2r} - 1&r < \frac{1}{2}\\
1 - \sqrt{2 - 2r}&r \geq \frac{1}{2}\\
\end{cases}$$

However in sRGB files we have to deal with gamma. Essentially we have an invertable transform $t$ (often $t(x) = x^{2.4}$) that is applied before/after quantization to allocate more steps to lower intensities.

This means that the average color when viewing is no longer (ignoring quantization for now):

$$\int\_0^1\left(s + n(s, r)\right)\text{d}r = s$$

But is instead:

$$\int\_0^1 t^{-1}\left(s + n(s, r)\right)\text{d}r > s$$

How can we change our noise function $n$ so that the average color is again $s$ after dithering, while maintaining our properties from the triangular distribution as much as possible?

For the record, the main property the triangular distribution gives is that the variance after rounding becomes constant:

$$\int\_0^1\left(\left[(s + n(s, r))(m-1)\right]/(m-1) - s\right)^2\text{d}r = c$$

Where $[x]$ denotes $x$ rounded to the closest integer. Note that this only holds in the triangular part of the domain of $n$, so when $\frac{1}{2(m-1)} \leq s \leq 1-\frac{1}{2(m-1)}$.

If you want to give a shot, here is a demo that can show you the error from dithering, which has gamma correction: <https://www.shadertoy.com/view/3tf3Dn>.

**Tags:** noise, gamma

> **Comments:**
> **username_1**: I think now I understand your question. I'm not familiar with the language you implemented the function you mentioned, could you maybe write down $n(s,r)$ explicitly in "math notation"? :) (btw: I assume $r$ is uniform on $[0,1]$, right?)  
> **username_0**: @username_1 It's only `dither_quantize` you need to look at. There `c` is $s$, `rng` is $r$, `depth` is $m$, `ci` is $s\cdot (m-1)$ and `d` is $n(s, r)\cdot (m-1)$. `a ? b : c` means `b` if `a` else `c`, and `clamp(x, lo, hi)` is `min(max(x, lo), hi)`. Finally `uint(x)` is basically $\lfloor x \rfloor$.  
> **username_0**: @username_1 One thing that isn't particularly clean here is that we must not forget $n(s, r)$ also depends on $m$, as fewer quantization steps means the noise must increase in size. It is $n(s, r) \cdot (m-1)$ that doesn't change as $m$ does.  
> **username_0**: @username_1 There is another way you can view it, which I posted here: https://math.stackexchange.com/questions/3200249/maintaining-probability-distribution-under-transform. I convolved the above noise function with a unit square to find the probability that a signal $s$ gets quantized to $[s]$ after adding noise. Then you can view the whole quantizing operation as summing a couple random variables (e.g. one for the chance at $[s] -1$, one for $[s]$ and one for $[s] + 1$).  

---

## 0 answers

