<p>Are there any way to optimize this flow:</p>
<ol>
<li>The frame captured from camera (NV21 byte array)</li>
<li>Convert this frame to I420 format</li>
<li>Upload Y data to <code>y_tex</code>; U data to <code>u_tex</code>; V data to <code>v_tex</code></li>
<li>Run the following fragment shader:</li>
</ol>
<pre><code>...
 float greenValue(vec2 coord)
 {
  return texture2D(y_tex, coord).r 
    - 0.344 * (texture2D(u_tex, coord).r - 0.5) 
    - 0.714 * (texture2D(v_tex, coord).r - 0.5);
 }
 // some code .... if (current pixel is skin color)
 vec2 blurCoordinates[20];
 blurCoordinates[0] = interp_tc.xy + singleStepOffset * vec2(0.0, -10.0);
 blurCoordinates[1] = interp_tc.xy + singleStepOffset * vec2(0.0, 10.0);
 blurCoordinates[2] = interp_tc.xy + singleStepOffset * vec2(-10.0, 0.0);
 blurCoordinates[3] = interp_tc.xy + singleStepOffset * vec2(10.0, 0.0);
 blurCoordinates[4] = interp_tc.xy + singleStepOffset * vec2(5.0, -8.0);
 blurCoordinates[5] = interp_tc.xy + singleStepOffset * vec2(5.0, 8.0);
 blurCoordinates[6] = interp_tc.xy + singleStepOffset * vec2(-5.0, 8.0);
 blurCoordinates[7] = interp_tc.xy + singleStepOffset * vec2(-5.0, -8.0);
 blurCoordinates[8] = interp_tc.xy + singleStepOffset * vec2(8.0, -5.0);
 blurCoordinates[9] = interp_tc.xy + singleStepOffset * vec2(8.0, 5.0);
 blurCoordinates[10] = interp_tc.xy + singleStepOffset * vec2(-8.0, 5.0);
 blurCoordinates[11] = interp_tc.xy + singleStepOffset * vec2(-8.0, -5.0);
 blurCoordinates[12] = interp_tc.xy + singleStepOffset * vec2(0.0, -6.0);
 blurCoordinates[13] = interp_tc.xy + singleStepOffset * vec2(0.0, 6.0);
 blurCoordinates[14] = interp_tc.xy + singleStepOffset * vec2(6.0, 0.0);
 blurCoordinates[15] = interp_tc.xy + singleStepOffset * vec2(-6.0, 0.0);
 blurCoordinates[16] = interp_tc.xy + singleStepOffset * vec2(-4.0, -4.0);
 blurCoordinates[17] = interp_tc.xy + singleStepOffset * vec2(-4.0, 4.0);
 blurCoordinates[18] = interp_tc.xy + singleStepOffset * vec2(4.0, -4.0);
 blurCoordinates[19] = interp_tc.xy + singleStepOffset * vec2(4.0, 4.0);
 // some code ....
 float sampleColor = centralColor.g * 20.0;
 sampleColor += greenValue(blurCoordinates[0]);
 sampleColor += greenValue(blurCoordinates[1]);
 sampleColor += greenValue(blurCoordinates[2]);
 sampleColor += greenValue(blurCoordinates[3]);
 sampleColor += greenValue(blurCoordinates[4]);
 sampleColor += greenValue(blurCoordinates[5]);
 sampleColor += greenValue(blurCoordinates[6]);
 sampleColor += greenValue(blurCoordinates[7]);
 sampleColor += greenValue(blurCoordinates[8]);
 sampleColor += greenValue(blurCoordinates[9]);
 sampleColor += greenValue(blurCoordinates[10]);
 sampleColor += greenValue(blurCoordinates[11]);
 sampleColor += greenValue(blurCoordinates[12]) * 2.0;
 sampleColor += greenValue(blurCoordinates[13]) * 2.0;
 sampleColor += greenValue(blurCoordinates[14]) * 2.0;
 sampleColor += greenValue(blurCoordinates[15]) * 2.0;
 sampleColor += greenValue(blurCoordinates[16]) * 2.0;
 sampleColor += greenValue(blurCoordinates[17]) * 2.0;
 sampleColor += greenValue(blurCoordinates[18]) * 2.0;
 sampleColor += greenValue(blurCoordinates[19]) * 2.0;
</code></pre>
<p><strong>Problem:</strong></p>
<p>I tested on weak device (galaxy A6) and capture with HD resolution frame (1280x720). It take ~100ms/frame. The main reason of this bottle neck is the large number of <code>texture2D()</code> call. Each frame make ~60,000,000 call <code>texture2D()</code>.</p>
<p><strong>Idea:</strong></p>
<p>When I replace (<code>u_tex</code>, <code>v_tex</code>) by <code>y_tex</code> for test, it's very fast. So, I think performance maybe improved by use only <strong>one texture</strong> instead of <strong>three textures</strong> (<code>y_tex, u_tex, v_tex</code>).</p>
<p><strong>Question:</strong></p>
<p>Does my approach is feasible? Are there any way to optimize this shader or improve entire  this flow?</p>

CC BY-SA 4.0
2023-02-07T11:34:41.373Z
Can you give us more detail about how your shader works, and how neighboring pixels are interacting with each other? Why are you using these blur coordinates and not raster (n x n) coordinates?
Thomas
CC BY-SA 4.0
2023-02-07T11:42:45.670Z
Storing all three values to one texture will improve your performance, because texture lookups are very slow. So Yes, this is feasible
Thomas
CC BY-SA 4.0
2023-02-08T02:48:19.380Z
Thanks @Thomas, this code in an old project and wrote by an ex-employees. So I don't understand entire algorithm in shader! This is full shader: https://pqviet.notion.site/6552db82a8e6472e8aad8ff018f250a6
Viète
CC BY-SA 4.0
2023-02-08T03:16:30.143Z
@Thomas, I think the source of shader got from this: https://github.com/MasayukiSuda/GPUVideo-android/issues/76
Viète
CC BY-SA 4.0
2023-02-08T04:30:24.890Z
@Thomas,  Because many of the texels fetched by one pixel will also be used by neighbouring pixels, so can I pre-compute and store all result of greenValue(blurCoordinate[i]) to some where like SSBO then reuse the result in SSBO?
Viète
CC BY-SA 4.0
2023-02-08T08:15:06.903Z
I am not sure, if SSBOs are faster than texture lookups. But: when changing the blur coordinates to a grid (n x n), neighboring pixels will read the same position. When this behavior is fulfilled, it may be faster to use compute shaders. Compute shaders can hold "shared" variables, which are faster to access. So each pixel only read one value from the texture and store this value to the "shared" variables. After reading, you need a barrier to ensure, that all pixels are read. Then you can work on the "shared" variables. Have in mind, that the workgroup size is limited (border cases).
Thomas
<glsl><opengl-es><android>
Bottle neck in fragment shader because large number of texture2D call
Viète
# Bottle neck in fragment shader because large number of texture2D call

(Asked by: username_0 on 2023-02-07)

Are there any way to optimize this flow:

1. The frame captured from camera (NV21 byte array)
2. Convert this frame to I420 format
3. Upload Y data to `y_tex`; U data to `u_tex`; V data to `v_tex`
4. Run the following fragment shader:

```
...
 float greenValue(vec2 coord)
 {
  return texture2D(y_tex, coord).r 
    - 0.344 * (texture2D(u_tex, coord).r - 0.5) 
    - 0.714 * (texture2D(v_tex, coord).r - 0.5);
 }
 // some code .... if (current pixel is skin color)
 vec2 blurCoordinates[20];
 blurCoordinates[0] = interp_tc.xy + singleStepOffset * vec2(0.0, -10.0);
 blurCoordinates[1] = interp_tc.xy + singleStepOffset * vec2(0.0, 10.0);
 blurCoordinates[2] = interp_tc.xy + singleStepOffset * vec2(-10.0, 0.0);
 blurCoordinates[3] = interp_tc.xy + singleStepOffset * vec2(10.0, 0.0);
 blurCoordinates[4] = interp_tc.xy + singleStepOffset * vec2(5.0, -8.0);
 blurCoordinates[5] = interp_tc.xy + singleStepOffset * vec2(5.0, 8.0);
 blurCoordinates[6] = interp_tc.xy + singleStepOffset * vec2(-5.0, 8.0);
 blurCoordinates[7] = interp_tc.xy + singleStepOffset * vec2(-5.0, -8.0);
 blurCoordinates[8] = interp_tc.xy + singleStepOffset * vec2(8.0, -5.0);
 blurCoordinates[9] = interp_tc.xy + singleStepOffset * vec2(8.0, 5.0);
 blurCoordinates[10] = interp_tc.xy + singleStepOffset * vec2(-8.0, 5.0);
 blurCoordinates[11] = interp_tc.xy + singleStepOffset * vec2(-8.0, -5.0);
 blurCoordinates[12] = interp_tc.xy + singleStepOffset * vec2(0.0, -6.0);
 blurCoordinates[13] = interp_tc.xy + singleStepOffset * vec2(0.0, 6.0);
 blurCoordinates[14] = interp_tc.xy + singleStepOffset * vec2(6.0, 0.0);
 blurCoordinates[15] = interp_tc.xy + singleStepOffset * vec2(-6.0, 0.0);
 blurCoordinates[16] = interp_tc.xy + singleStepOffset * vec2(-4.0, -4.0);
 blurCoordinates[17] = interp_tc.xy + singleStepOffset * vec2(-4.0, 4.0);
 blurCoordinates[18] = interp_tc.xy + singleStepOffset * vec2(4.0, -4.0);
 blurCoordinates[19] = interp_tc.xy + singleStepOffset * vec2(4.0, 4.0);
 // some code ....
 float sampleColor = centralColor.g * 20.0;
 sampleColor += greenValue(blurCoordinates[0]);
 sampleColor += greenValue(blurCoordinates[1]);
 sampleColor += greenValue(blurCoordinates[2]);
 sampleColor += greenValue(blurCoordinates[3]);
 sampleColor += greenValue(blurCoordinates[4]);
 sampleColor += greenValue(blurCoordinates[5]);
 sampleColor += greenValue(blurCoordinates[6]);
 sampleColor += greenValue(blurCoordinates[7]);
 sampleColor += greenValue(blurCoordinates[8]);
 sampleColor += greenValue(blurCoordinates[9]);
 sampleColor += greenValue(blurCoordinates[10]);
 sampleColor += greenValue(blurCoordinates[11]);
 sampleColor += greenValue(blurCoordinates[12]) * 2.0;
 sampleColor += greenValue(blurCoordinates[13]) * 2.0;
 sampleColor += greenValue(blurCoordinates[14]) * 2.0;
 sampleColor += greenValue(blurCoordinates[15]) * 2.0;
 sampleColor += greenValue(blurCoordinates[16]) * 2.0;
 sampleColor += greenValue(blurCoordinates[17]) * 2.0;
 sampleColor += greenValue(blurCoordinates[18]) * 2.0;
 sampleColor += greenValue(blurCoordinates[19]) * 2.0;

```

**Problem:**

I tested on weak device (galaxy A6) and capture with HD resolution frame (1280x720). It take ~100ms/frame. The main reason of this bottle neck is the large number of `texture2D()` call. Each frame make ~60,000,000 call `texture2D()`.

**Idea:**

When I replace (`u_tex`, `v_tex`) by `y_tex` for test, it's very fast. So, I think performance maybe improved by use only **one texture** instead of **three textures** (`y_tex, u_tex, v_tex`).

**Question:**

Does my approach is feasible? Are there any way to optimize this shader or improve entire this flow?

**Tags:** glsl, opengl-es, android

> **Comments:**
> **username_1**: Can you give us more detail about how your shader works, and how neighboring pixels are interacting with each other? Why are you using these blur coordinates and not raster (n x n) coordinates?  
> **username_1**: Storing all three values to one texture will improve your performance, because texture lookups are very slow. So Yes, this is feasible  
> **username_0**: Thanks @username_1, this code in an old project and wrote by an ex-employees. So I don't understand entire algorithm in shader! This is full shader: https://pqviet.notion.site/6552db82a8e6472e8aad8ff018f250a6  
> **username_0**: @username_1, I think the source of shader got from this: https://github.com/MasayukiSuda/GPUVideo-android/issues/76  
> **username_0**: @username_1,  Because many of the texels fetched by one pixel will also be used by neighbouring pixels, so can I pre-compute and store all result of greenValue(blurCoordinate[i]) to some where like SSBO then reuse the result in SSBO?  
> **username_1**: I am not sure, if SSBOs are faster than texture lookups. But: when changing the blur coordinates to a grid (n x n), neighboring pixels will read the same position. When this behavior is fulfilled, it may be faster to use compute shaders. Compute shaders can hold "shared" variables, which are faster to access. So each pixel only read one value from the texture and store this value to the "shared" variables. After reading, you need a barrier to ensure, that all pixels are read. Then you can work on the "shared" variables. Have in mind, that the workgroup size is limited (border cases).  

---

## 0 answers

