<p>From the way the problem only manifests when you start overwriting previously used texture tiles, I suspect you're running afoul of some driver heuristic that's attempting to manage the lifetime of the texture data on the GPU.</p>

<p>In general, when you update buffer or texture data from the CPU, the driver has to make sure it doesn't overwrite anything that might still be in use by the GPU (considering that the CPU and GPU run asynchronously to each other). There are various ways the driver can do this, such as internally double-buffering the texture, inserting sync points (for the CPU to wait on the GPU), or initially copying data to a temporary staging buffer and then later re-copying it to its final destination. All this stuff is under-the-hood and the driver makes its own decisions about what strategy to employ in any given situation, using some heuristics based on texture size, usage patterns, etc.</p>

<p>I can only guess at the details, but I suspect what's happening is that the driver is getting into a state where it thinks those previously-used texture tiles are still in use on the GPU somehow. Then when you start to overwrite them, it decides that it needs to make a second instance of the whole texture, and laboriously copies everything over before finally doing the updates you wanted—or something like that. That could be the many tiny draws you see in Nsight.</p>

<p>It's still a mystery why it's taking tens of milliseconds to do that, though. Maybe the textures are large enough that a second copy of them causes you to run out of VRAM? In that case, you'd get a lot of thrashing as things are spilled to main RAM. That's just a guess, though. As for why the bad behavior only happens for a few tiles and then goes back to normal, it could be that the driver heuristics are deciding to switch to a different strategy at that point.</p>

<p>Anyway, as for how to fix it, you might try using <a href="http://docs.gl/gl4/glInvalidateTexSubImage" rel="noreferrer"><code>glInvalidateTexSubImage</code></a> whenever you free a texture tile. This lets the driver know that you don't care what was in that texture tile before, which <em>hopefully</em> will convince it to just update the texture in-place rather than doing anything crazy with it.</p>

<p>Another thing to try is to make sure you don't try to re-use a texture tile in the very next frame after you free it, but give it a frame or two of "downtime" before re-using it. But it sounds like that's probably the case already, given that you prefer unused tiles before you start re-using them.</p>

<p>If that doesn't help, I'd recommend sending an email to devsupport@nvidia.com. The OpenGL driver team there can likely take your Nsight captures and help diagnose the issue, and fix it if it's due to a driver bug.</p>

CC BY-SA 3.0
2017-04-11T00:34:59.973Z
That makes sense, though I doubt I'm running out of VRAM honestly because last time I checked these textures are only ~40MB each or so and a 960M has 4GB so there should plenty. Also yeah that whole prioritize uninitialized and first in/first out system of managing the tiles should always make sure it updates the oldest freed tile. I'll try that call though to see if it works.
Lemon Drop
CC BY-SA 3.0
2017-04-11T17:58:25.537Z
Well I implemented the `glInvalidateTexSubImage` thing when the tiles are freed (assuming I did it right, it calls it on each mipmap level for the layer in question) and it doesn't seem to change anything. I'm going to keep it because it probably will be helping the GPU in some way but I guess sending something to NVIDIA might be the best option now (I'll likely try to make a minimal example of it in a small program later when I have time). For now I'm also going to try initializing the entire texture upfront to see if that helps.
Lemon Drop
CC BY-SA 3.0
2017-04-14T06:24:35.600Z
You could also try to use PBOs to map memory client side from the driver. Then you'd fill up the PBOs, unmap the memory, and queue up a `glCompressedTexSubImage2D` to let the driver do it's own upload.
Mokosha
CC BY-SA 3.0
2017-04-14T06:36:44.987Z
@Mokosha Yeah I am probably going to do that for my next project, I was just not aware that was a thing at the time of making this and due to time constraints I can't exactly implement it now.
Lemon Drop
CC BY-SA 3.0
2017-04-11T00:13:55.873Z
2017-04-11T00:13:55.873Z
Nathan Reed
<p>Recently I have been having a rather confusing performance drop with loading in layers of a 2D texture array, but only in one case from what I can tell, leaving me to speculate on why it is happening as I have never seen anything like this before.</p>

<p>Firstly to explain some context for this problem, I implemented a simple version of a virtual texture atlas based on some papers I read by NVIDIA and idTech which in short loads textures (which have been split into fixed size tiles) into various levels of the 2D texture array based on which layers are considered "free" in a CPU-side list of free indices. Sampling in the fragment shader is then preformed by accessing a page table stored as a persistently coherently mapped SSBO (for CPU-side updating) which associates offsets in the UVs with layers of the texture array.</p>

<p>Due to the fact that the texture when created is uninitialized, every layer is considered free, so texture tiles to be loaded will slowly fill up the uninitialized layers (preferring them over previously freed layers). Once all of the uninitialized layers have been used, additional tiles will then begin overwriting previously freed layers, if any exist when needed. Also note that I am using <code>glCompressedTexSubImage3D</code> to load in each layer and that I load in mipmap levels as well manually per layer with this call as well.</p>

<p>This system works fine and was tested rather extensively until I began to notice the problem which this question is about. In short, after all of the uninitialized layers of the texture array were used up, the next 5-15 (I don't know exactly how many, but it's just a few) texture tiles which are loaded suffer massive performance drops. After that, everything returns to normal and the problem never returns (from what I can tell). Also note that this is while some currently drawn textures were also in use at the lower end of the texture array which is right before where the first free tiles begin (due to the first in first out nature of the free list).</p>

<p>Upon examining the problem further, I noticed through using the NVIDIA Nsight graphics profiling tool that the call after <code>glCompressedTexSubImage3D</code> in the cases where performance was impacted blocks on the CPU for very long periods of time, usually 10-40ms per mipmap level (in this case it was <code>glGetError</code>, but any other call following it had the same effect after disabling error checking to ensure that was not the problem).</p>

<p>Since the API was blocking, I assumed that it was some sort of synchronization issue with the GPU perhaps due to the locality of the newly loaded tiles with respect to tiles currently being drawn as it may have considered them "in use". That hypothesis did not add up in my mind due to how long the calls were blocking for as each one was the length of multiple frames at 16ms as well as how this problem never appears to return even after multiple times through this same process, though I'm not ignoring that possibility.</p>

<p>Another odd thing I found was that during the time the next call was blocking, many tiny GPU commands were being executed according to NVIDIA Nsight, which I usually only see under any sort of drawing activity.</p>

<p>At this point I'm pretty much stumped and don't know what could possibly be causing this, so any help would be greatly appreciated. Additional details are provided below (and more can be requested if needed):</p>

<p><strong>Texture Array</strong> (there are 3 of them):</p>

<ul>
<li>Format: <code>COMPRESSED_RGB_BPTC_UNSIGNED_FLOAT</code>, <code>COMPRESSED_RED_RGTC1</code>,
<code>COMPRESSED_RG_RGTC2</code></li>
<li>Extents: 256x256x512, 9 mipmap levels</li>
</ul>

<p><strong>GPU</strong>:</p>

<p>GeForce GTX 960M</p>

<p><strong>NVIDIA Nsight Results</strong>:</p>

<p>Wide view of 3 long frames where the slow texture loading happens (the line down the screen is centered on the first texture loading call):</p>

<p><a href="https://i.sstatic.net/nwjN4.png" rel="noreferrer"><img src="https://i.sstatic.net/nwjN4.png" alt="Overall view"></a></p>

<p>Zoomed in view near the very start of the first slow call to <code>glCompressedTexSubImage3D</code> (which cannot be seen as it only blocks for a few microseconds), note the many tiny green lines below the blocking <code>glGetError</code> calls representing GPU commands (as again the line down the screen is centered on the first texture loading call):</p>

<p><a href="https://i.sstatic.net/SfLfV.png" rel="noreferrer"><img src="https://i.sstatic.net/SfLfV.png" alt="Zoomed view"></a></p>

CC BY-SA 3.0
2017-04-12T09:04:46.847Z
Could it be some kind of false sharing: the memory you're overwriting with the texture upload shares cache lines with parts of the atlas that the GPU is reading at the same time? What size & shape is the region of the atlas you're overwriting?
Dan Hulme
CC BY-SA 3.0
2017-04-12T16:26:47.467Z
@DanHulme Since its a 2D Array Texture I just load in each mipmap level of a layer I want to overwrite. I have 50 or so layers of the 512 layer texture in use by the GPU and the layers which cause the odd performance issues when loading are right next to them since while finding new layers it wraps around from the end after walking through all the uninitalized layers back to the oldest "free" page in a list, which happens to be right next to that active range of texture layers. I thought it had something to do with that but the fact that it only happens the first time made me think otherwise.
Lemon Drop
CC BY-SA 3.0
2017-04-12T16:34:47.830Z
Also I'm going to attempt to make a minimal example of this problem once I have some time to as I am making a lot of assumptions most likely about what is going on, not to mention all of the other things going on in this project which could be affecting this texture loading (there's a lot of places where I use `glClientWaitSync` and whatnot for synchronization but I doubt those would cause issues with something like this).
Lemon Drop
<opengl><texture><performance>
Performance impact during texture layer loading
Lemon Drop
# Performance impact during texture layer loading

(Asked by: username_0 on 2017-04-10)

Recently I have been having a rather confusing performance username_0 with loading in layers of a 2D texture array, but only in one case from what I can tell, leaving me to speculate on why it is happening as I have never seen anything like this before.

Firstly to explain some context for this problem, I implemented a simple version of a virtual texture atlas based on some papers I read by NVIDIA and idTech which in short loads textures (which have been split into fixed size tiles) into various levels of the 2D texture array based on which layers are considered "free" in a CPU-side list of free indices. Sampling in the fragment shader is then preformed by accessing a page table stored as a persistently coherently mapped SSBO (for CPU-side updating) which associates offsets in the UVs with layers of the texture array.

Due to the fact that the texture when created is uninitialized, every layer is considered free, so texture tiles to be loaded will slowly fill up the uninitialized layers (preferring them over previously freed layers). Once all of the uninitialized layers have been used, additional tiles will then begin overwriting previously freed layers, if any exist when needed. Also note that I am using `glCompressedTexSubImage3D` to load in each layer and that I load in mipmap levels as well manually per layer with this call as well.

This system works fine and was tested rather extensively until I began to notice the problem which this question is about. In short, after all of the uninitialized layers of the texture array were used up, the next 5-15 (I don't know exactly how many, but it's just a few) texture tiles which are loaded suffer massive performance drops. After that, everything returns to normal and the problem never returns (from what I can tell). Also note that this is while some currently drawn textures were also in use at the lower end of the texture array which is right before where the first free tiles begin (due to the first in first out nature of the free list).

Upon examining the problem further, I noticed through using the NVIDIA Nsight graphics profiling tool that the call after `glCompressedTexSubImage3D` in the cases where performance was impacted blocks on the CPU for very long periods of time, usually 10-40ms per mipmap level (in this case it was `glGetError`, but any other call following it had the same effect after disabling error checking to ensure that was not the problem).

Since the API was blocking, I assumed that it was some sort of synchronization issue with the GPU perhaps due to the locality of the newly loaded tiles with respect to tiles currently being drawn as it may have considered them "in use". That hypothesis did not add up in my mind due to how long the calls were blocking for as each one was the length of multiple frames at 16ms as well as how this problem never appears to return even after multiple times through this same process, though I'm not ignoring that possibility.

Another odd thing I found was that during the time the next call was blocking, many tiny GPU commands were being executed according to NVIDIA Nsight, which I usually only see under any sort of drawing activity.

At this point I'm pretty much stumped and don't know what could possibly be causing this, so any help would be greatly appreciated. Additional details are provided below (and more can be requested if needed):

**Texture Array** (there are 3 of them):

* Format: `COMPRESSED_RGB_BPTC_UNSIGNED_FLOAT`, `COMPRESSED_RED_RGTC1`,
  `COMPRESSED_RG_RGTC2`
* Extents: 256x256x512, 9 mipmap levels

**GPU**:

GeForce GTX 960M

**NVIDIA Nsight Results**:

Wide view of 3 long frames where the slow texture loading happens (the line down the screen is centered on the first texture loading call):

[![Overall view](https://i.sstatic.net/nwjN4.png)](https://i.sstatic.net/nwjN4.png)

Zoomed in view near the very start of the first slow call to `glCompressedTexSubImage3D` (which cannot be seen as it only blocks for a few microseconds), note the many tiny green lines below the blocking `glGetError` calls representing GPU commands (as again the line down the screen is centered on the first texture loading call):

[![Zoomed view](https://i.sstatic.net/SfLfV.png)](https://i.sstatic.net/SfLfV.png)

**Tags:** opengl, texture, performance

> **Comments:**
> **username_1**: Could it be some kind of false sharing: the memory you're overwriting with the texture upload shares cache lines with parts of the atlas that the GPU is reading at the same time? What size & shape is the region of the atlas you're overwriting?  
> **username_0**: @username_1 Since its a 2D Array Texture I just load in each mipmap level of a layer I want to overwrite. I have 50 or so layers of the 512 layer texture in use by the GPU and the layers which cause the odd performance issues when loading are right next to them since while finding new layers it wraps around from the end after walking through all the uninitalized layers back to the oldest "free" page in a list, which happens to be right next to that active range of texture layers. I thought it had something to do with that but the fact that it only happens the first time made me think otherwise.  
> **username_0**: Also I'm going to attempt to make a minimal example of this problem once I have some time to as I am making a lot of assumptions most likely about what is going on, not to mention all of the other things going on in this project which could be affecting this texture loading (there's a lot of places where I use `glClientWaitSync` and whatnot for synchronization but I doubt those would cause issues with something like this).  

---

## 1 answers

### Answer by username_2 on 2017-04-11. Score: 5

From the way the problem only manifests when you start overwriting previously used texture tiles, I suspect you're running afoul of some driver heuristic that's attempting to manage the lifetime of the texture data on the GPU.

In general, when you update buffer or texture data from the CPU, the driver has to make sure it doesn't overwrite anything that might still be in use by the GPU (considering that the CPU and GPU run asynchronously to each other). There are various ways the driver can do this, such as internally double-buffering the texture, inserting sync points (for the CPU to wait on the GPU), or initially copying data to a temporary staging buffer and then later re-copying it to its final destination. All this stuff is under-the-hood and the driver makes its own decisions about what strategy to employ in any given situation, using some heuristics based on texture size, usage patterns, etc.

I can only guess at the details, but I suspect what's happening is that the driver is getting into a state where it thinks those previously-used texture tiles are still in use on the GPU somehow. Then when you start to overwrite them, it decides that it needs to make a second instance of the whole texture, and laboriously copies everything over before finally doing the updates you wanted—or something like that. That could be the many tiny draws you see in Nsight.

It's still a mystery why it's taking tens of milliseconds to do that, though. Maybe the textures are large enough that a second copy of them causes you to run out of VRAM? In that case, you'd get a lot of thrashing as things are spilled to main RAM. That's just a guess, though. As for why the bad behavior only happens for a few tiles and then goes back to normal, it could be that the driver heuristics are deciding to switch to a different strategy at that point.

Anyway, as for how to fix it, you might try using [`glInvalidateTexSubImage`](http://docs.gl/gl4/glInvalidateTexSubImage) whenever you free a texture tile. This lets the driver know that you don't care what was in that texture tile before, which *hopefully* will convince it to just update the texture in-place rather than doing anything crazy with it.

Another thing to try is to make sure you don't try to re-use a texture tile in the very next frame after you free it, but give it a frame or two of "downtime" before re-using it. But it sounds like that's probably the case already, given that you prefer unused tiles before you start re-using them.

If that doesn't help, I'd recommend sending an email to devsupport@nvidia.com. The OpenGL driver team there can likely take your Nsight captures and help diagnose the issue, and fix it if it's due to a driver bug.

> **Comments:**
> **username_0**: That makes sense, though I doubt I'm running out of VRAM honestly because last time I checked these textures are only ~40MB each or so and a 960M has 4GB so there should plenty. Also yeah that whole prioritize uninitialized and first in/first out system of managing the tiles should always make sure it updates the oldest freed tile. I'll try that call though to see if it works.  
> **username_0**: Well I implemented the `glInvalidateTexSubImage` thing when the tiles are freed (assuming I did it right, it calls it on each mipmap level for the layer in question) and it doesn't seem to change anything. I'm going to keep it because it probably will be helping the GPU in some way but I guess sending something to NVIDIA might be the best option now (I'll likely try to make a minimal example of it in a small program later when I have time). For now I'm also going to try initializing the entire texture upfront to see if that helps.  
> **username_3**: You could also try to use PBOs to map memory client side from the driver. Then you'd fill up the PBOs, unmap the memory, and queue up a `glCompressedTexSubImage2D` to let the driver do it's own upload.  
> **username_0**: @username_3 Yeah I am probably going to do that for my next project, I was just not aware that was a thing at the time of making this and due to time constraints I can't exactly implement it now.  

---

