<p>So it sounds like the main issue you're having is how to render something when all resources are already on the GPU. I would do it like this:</p>

<ol>
<li>Create the destination RGB texture that's the proper width and height</li>
<li>Create an FBO</li>
<li>Set the texture created in step 1 as the output (write) texture of the FBO</li>
<li>Bind the NV12 texture to the texture unit you'll use in your shader</li>
<li>Draw a textured quad that covers the entire output</li>
<li>Apply a shader that samples the NV12 texture to generate the RGB value at each fragment</li>
</ol>

<p>For step 1 you use <code>glGenTextures()</code> and <code>glTexImage2D()</code>.</p>

<p>For steps 2 and 3 you use <code>glGenFramebuffers()</code> and <code>glBindFramebuffer()</code> passing <code>glBindFramebuffer()</code> the texture created in step 1 to attach it to <code>GL_COLOR_ATTACHMENT0</code>. (See <a href="https://www.khronos.org/opengl/wiki/Framebuffer_Object" rel="nofollow noreferrer">here</a> for more details.)</p>

<p>For step 4 you use <code>glBindTexture()</code>.</p>

<p>For step 5 you can simply draw a quad that goes from (-1, -1) to (1, 1). You don't need any model view matrix or anything.</p>

<p>For step 6 you'll need to calculate where the red, green, and blue samples that correspond to the current output pixel are in the input texture and then sample them using the <code>texture()</code> function. You'll need to do 3 different look-ups for each output pixel. <a href="http://paulbourke.net/dataformats/nv12/" rel="nofollow noreferrer">This link has C code</a> for figuring out where each sample is from. </p>

<p>Once you're done you can delete the frame buffer and use the texture created in step 1 in your next step (display it, process it, save it to disk, or whatever).</p>

CC BY-SA 4.0
2019-07-28T06:27:26.567Z
Thanks, I'm learning about frameBuffers and I kinda understood all you said. I think I can follow it after some days of study. There's just one thing that I didn't understand. After calling `ret = avcodec_receive_frame(avctx, frame);`, ffmpeg stores my decoded frame somewhere on GPU and has a pointer to it called frame, on CPU. How do I make this `frame` get into my OpenGL program? I believe this is what you meant by `Bind the NV12 texture to the texture unit you'll use in your shader`
Guerlando OCs
CC BY-SA 4.0
2019-07-28T14:51:26.820Z
I've not used ffmpeg, so I don't know for sure, but if you want to avoid going from the CPU back to the GPU you'll need to get the texture ID from ffmpeg. Otherwise you can upload the texture from the CPU using `glTexImage2D()`.
user1118321
CC BY-SA 4.0
2019-07-27T23:23:40.850Z
2019-07-27T23:23:40.850Z
user1118321
<p>Decoding video will result in different pixel formats on different GPUs and drivers. I have an application which can render (using OpenGL), only the RGB8 pixel format, therefore I need to do color conversion from the decoded pixel format to RGB8. I can access the gl... functions in this program but I cannot edit the shader, so I'm stuck with RGB8.</p>

<p>This ffmpeg example demonstrates how to do hardware decoding: <a href="https://github.com/FFmpeg/FFmpeg/blob/release/4.2/doc/examples/hw_decode.c" rel="nofollow noreferrer">https://github.com/FFmpeg/FFmpeg/blob/release/4.2/doc/examples/hw_decode.c</a></p>

<p>At line 109 it does this:</p>

<pre><code>/* retrieve data from GPU to CPU */
            if ((ret = av_hwframe_transfer_data(sw_frame, frame, 0)) &lt; 0) {
</code></pre>

<p>I want to avoid this because it takes time. Therefore, I need a way to reuse that decoded video, which is in GPU memory, to redo color conversion.</p>

<p>Is it possible to create a shader program that takes input from the GPU memory, not CPU memory? If I want to pass image from CPU to my shader, I simply use <code>glTexImage2D</code>. Is there a similar way to do it but from GPU memory?</p>

<p>Can I do even deeper and render the color decoded video on screen without copying from GPU memory too?</p>

<p>If what I wanna do is possible, are there shader programs available to color convert pixels for some major graphics cards/drivers? I don't want to support all of them, but I'd like to support at least the Jetson Nano board from NVIDIA.</p>

CC BY-SA 4.0
2019-07-27T03:41:55.587Z
Can you be more specific? Are you looking to convert from something like Y'CbCr to RGB (or the opposite)? Or HSV or La*b*? It's certainly possible to convert between color spaces on the GPU without going to the CPU, but it's not clear what you're asking.
user1118321
CC BY-SA 4.0
2019-07-27T04:03:25.480Z
@user1118321 hi, thank you for your answer. Well, I heard that each GPU decodes to a different color format. So what I need is to convert from this unknown color format to RGB8. I know it's a hard task to do it for all color formats, but I plan to support, for now, only the format outputted from the Jetson Nano board from NVIDIA. I don't know its output because the ffmpeg support for this board is not ready yet but I guess it's NV12 (the color format for CUDA). Can you tell more about how to do these color conversions?
Guerlando OCs
CC BY-SA 4.0
2019-07-27T21:25:38.563Z
[This article on NV12](http://paulbourke.net/dataformats/nv12/) says, "The NV12 image format is commonly found as the native format from various machine vision, and other, video cameras." The article has CPU code for converting from NV12 to Y'CbCr and then to RGB. I don't see why you couldn't modify that code to work on the GPU.
user1118321
CC BY-SA 4.0
2019-07-27T22:37:21.833Z
@user1118321 thanks, that's a starting point. But how should I do this conversion in GPU? I have no idea, and I searched about rendering from GPU memory but couldn't find anything. Could you give me a starting point?
Guerlando OCs
CC BY-SA 4.0
2019-07-27T23:23:53.340Z
OK, I've added an answer that goes into more detail.
user1118321
<opengl><gpu>
Is it possible to do color conversion on decoded video without getting it from GPUmemory to CPU memory?
Guerlando OCs
# Is it possible to do color conversion on decoded video without getting it from GPUmemory to CPU memory?

(Asked by: username_0 on 2019-07-27)

Decoding video will result in different pixel formats on different GPUs and drivers. I have an application which can render (using OpenGL), only the RGB8 pixel format, therefore I need to do color conversion from the decoded pixel format to RGB8. I can access the gl... functions in this program but I cannot edit the shader, so I'm stuck with RGB8.

This ffmpeg example demonstrates how to do hardware decoding: <https://github.com/FFmpeg/FFmpeg/blob/release/4.2/doc/examples/hw_decode.c>

At line 109 it does this:

```
/* retrieve data from GPU to CPU */
            if ((ret = av_hwframe_transfer_data(sw_frame, frame, 0)) < 0) {

```

I want to avoid this because it takes time. Therefore, I need a way to reuse that decoded video, which is in GPU memory, to redo color conversion.

Is it possible to create a shader program that takes input from the GPU memory, not CPU memory? If I want to pass image from CPU to my shader, I simply use `glTexImage2D`. Is there a similar way to do it but from GPU memory?

Can I do even deeper and render the color decoded video on screen without copying from GPU memory too?

If what I wanna do is possible, are there shader programs available to color convert pixels for some major graphics cards/drivers? I don't want to support all of them, but I'd like to support at least the Jetson Nano board from NVIDIA.

**Tags:** opengl, gpu

> **Comments:**
> **username_1**: Can you be more specific? Are you looking to convert from something like Y'CbCr to RGB (or the opposite)? Or HSV or La*b*? It's certainly possible to convert between color spaces on the GPU without going to the CPU, but it's not clear what you're asking.  
> **username_0**: @username_1 hi, thank you for your answer. Well, I heard that each GPU decodes to a different color format. So what I need is to convert from this unknown color format to RGB8. I know it's a hard task to do it for all color formats, but I plan to support, for now, only the format outputted from the Jetson Nano board from NVIDIA. I don't know its output because the ffmpeg support for this board is not ready yet but I guess it's NV12 (the color format for CUDA). Can you tell more about how to do these color conversions?  
> **username_1**: [This article on NV12](http://paulbourke.net/dataformats/nv12/) says, "The NV12 image format is commonly found as the native format from various machine vision, and other, video cameras." The article has CPU code for converting from NV12 to Y'CbCr and then to RGB. I don't see why you couldn't modify that code to work on the GPU.  
> **username_0**: @username_1 thanks, that's a starting point. But how should I do this conversion in GPU? I have no idea, and I searched about rendering from GPU memory but couldn't find anything. Could you give me a starting point?  
> **username_1**: OK, I've added an answer that goes into more detail.  

---

## 1 answers

### Answer by username_1 on 2019-07-27. Score: 2

So it sounds like the main issue you're having is how to render something when all resources are already on the GPU. I would do it like this:

1. Create the destination RGB texture that's the proper width and height
2. Create an FBO
3. Set the texture created in step 1 as the output (write) texture of the FBO
4. Bind the NV12 texture to the texture unit you'll use in your shader
5. Draw a textured quad that covers the entire output
6. Apply a shader that samples the NV12 texture to generate the RGB value at each fragment

For step 1 you use `glGenTextures()` and `glTexImage2D()`.

For steps 2 and 3 you use `glGenFramebuffers()` and `glBindFramebuffer()` passing `glBindFramebuffer()` the texture created in step 1 to attach it to `GL_COLOR_ATTACHMENT0`. (See [here](https://www.khronos.org/opengl/wiki/Framebuffer_Object) for more details.)

For step 4 you use `glBindTexture()`.

For step 5 you can simply draw a quad that goes from (-1, -1) to (1, 1). You don't need any model view matrix or anything.

For step 6 you'll need to calculate where the red, green, and blue samples that correspond to the current output pixel are in the input texture and then sample them using the `texture()` function. You'll need to do 3 different look-ups for each output pixel. [This link has C code](http://paulbourke.net/dataformats/nv12/) for figuring out where each sample is from.

Once you're done you can delete the frame buffer and use the texture created in step 1 in your next step (display it, process it, save it to disk, or whatever).

> **Comments:**
> **username_0**: Thanks, I'm learning about frameBuffers and I kinda understood all you said. I think I can follow it after some days of study. There's just one thing that I didn't understand. After calling `ret = avcodec_receive_frame(avctx, frame);`, ffmpeg stores my decoded frame somewhere on GPU and has a pointer to it called frame, on CPU. How do I make this `frame` get into my OpenGL program? I believe this is what you meant by `Bind the NV12 texture to the texture unit you'll use in your shader`  
> **username_1**: I've not used ffmpeg, so I don't know for sure, but if you want to avoid going from the CPU back to the GPU you'll need to get the texture ID from ffmpeg. Otherwise you can upload the texture from the CPU using `glTexImage2D()`.  

---

