<p>Before we begin, it's important to understand that raw spec numbers are typically not meant for use by actual graphics programmers. They are marketing tools to sell products. Sure, they have to be &quot;accurate&quot; (somewhat), but that does not mean that they are meaningful.</p>
<p>As for what bandwidth actually means, consider two parking lots. They can hold a number of cars. Connecting them is a road. This road has a number of lanes. In terms of meaningful &quot;bandwidth,&quot; that is the answer to the question, &quot;how many cars can I get from one lot to the other?&quot; In our analogy, parking lots are pools of memory; the road is the connection between them.</p>
<p>So if someone tells you they have a 16 lane highway, does that mean something? Not really, not unless you know what parking lots that highway <em>connects</em>. The same goes for bandwidth; a raw number means nothing without knowing the source and destination memory for that bandwidth.</p>
<p>Marketers sometimes just add bandwidths from completely different memory pools together, as though the sum of lanes from 4 unconnected highways has some actual meaning.</p>
<p>So bandwidth is a measure of the rate of flow between pools of memory. It is the amount of data that can flow from one pool to another in a unit of time. These pools can be general GPU RAM, storage within a compute unit, various caches, etc. Many modern GPUs have memory architectures that connect many of these pools onto a single bus (effectively linking all parking lots by a single set of roads).</p>
<p>The other thing to remember is that pools of memory are not always <em>permanent</em>. Caches, compute storage, and the like are volatile; they get overwritten constantly by design. They exist because accessing bulk storage is slow, so operations that need to access memory quickly need small pools of faster memory closer to their use of this data.</p>
<p>If the bandwidth from GPU memory to a texture cache is 1'555GB/sec, this means that, within a 60fps frame, the total amount of storage that all shaders can access via texture fetches is 25.9GB. You may note that this is much smaller than the 40GB of storage such a GPU may have.</p>
<p>GPU bandwidths are so large <em>per second</em> because GPUs have to use that bandwidth many times per second. Once you look at them per frame, they're not so large.</p>
<p>Also, all bandwidth from one pool to another is purely <em>theoretical</em>. If nobody is reading from memory at any particular cycle, that cycle's worth of bandwidth goes <em>unused</em>. It can't be stored and use later when there's more memory pressure. You either use it or you lose it. And since GPU workloads tend to read memory in spurts, the <em>usable</em> bandwidth is often lower than the numerically available bandwidth.</p>

CC BY-SA 4.0
2022-05-27T07:11:22.917Z
So do you agree with Nathan reed that "memory bandwidth" is the bandwidth between gpu memory and gpu L2 cache?
user56834
CC BY-SA 4.0
2022-05-27T13:20:00.237Z
@user56834: No. "Memory bandwidth" as a term only makes sense in relation to the connection between two pools of memory. GPUs have *lots* of pools of memory. So the term "memory bandwidth" with no additional context is at best underspecified and should only be thought of as a general marketing buzzword, not a useful construct. They may have added together many actual bandwidth numbers to compute it. You don't know.
Nicol Bolas
CC BY-SA 4.0
2022-05-27T13:20:47.733Z
@user56834: Also, GPUs have *many* caches.
Nicol Bolas
CC BY-SA 4.0
2022-05-26T15:51:09.133Z
2022-05-26T15:51:09.133Z
Nicol Bolas
<p>Memory Bandwidth is the theoretical maximum amount of data that the bus can handle at any given time, playing a determining role in how quickly a GPU can access and utilize its frame buffer.</p>

CC BY-SA 4.0
2022-05-26T15:49:51.927Z
2022-05-26T15:49:51.927Z
Lin Kyūketsuki
<p>This question is motivated by looking at two numbers for different GPU's: (1) the memory size and (2) the memory bandwidth.</p>
<p>My understanding is that memory bandwidth means, the amount of data that can be copied from the system RAM to the GPU RAM (or vice versa) per second.</p>
<p>But looking at typical GPU's, the memory bandwitdh per second is much larger than the memory size: e.g. the Nvidia A100 has memory size 40 or 80 GB, and the memory bandwitdh is 1555 GB/s. This suggests that the GPU can copy its entire memory contents over to system RAM or back within about 1/20th or 1/40th of a second. Maybe this is true, but it just sounds extremely fast.</p>

CC BY-SA 4.0
2022-05-26T14:35:52.810Z
memory bandwidth is how fast the GPU processor can access *its own* memory
Esther
CC BY-SA 4.0
2022-05-26T15:07:58.140Z
So what exactly does it measure? does it measure the amount of data that can be transferred between the GPU RAM and the registers in the streaming multiprocessors?
user56834
CC BY-SA 4.0
2022-05-26T21:52:18.820Z
It's probably something like: the theoretical max rate at which data can be transferred from the GPU RAM into the GPU's L2 cache (or vice versa). This would set an upper limit on how quickly a rendering pass can run that needs to read a large amount of texture data (more than can fit into L2).

BTW, the bandwidth between GPU RAM and system RAM (for discrete GPU setups) is limited by the PCIe link bandwidth, which for PCIe Gen4 x16 is ~32 GB/s theoretical max (in practice somewhat less).
Nathan Reed
<gpu>
What does memory bandwidth of a GPU mean exactly?
user56834
# What does memory bandwidth of a GPU mean exactly?

(Asked by: username_0 on 2022-05-26)

This question is motivated by looking at two numbers for different GPU's: (1) the memory size and (2) the memory bandwidth.

My understanding is that memory bandwidth means, the amount of data that can be copied from the system RAM to the GPU RAM (or vice versa) per second.

But looking at typical GPU's, the memory bandwitdh per second is much larger than the memory size: e.g. the Nvidia A100 has memory size 40 or 80 GB, and the memory bandwitdh is 1555 GB/s. This suggests that the GPU can copy its entire memory contents over to system RAM or back within about 1/20th or 1/40th of a second. Maybe this is true, but it just sounds extremely fast.

**Tags:** gpu

> **Comments:**
> **username_1**: memory bandwidth is how fast the GPU processor can access *its own* memory  
> **username_0**: So what exactly does it measure? does it measure the amount of data that can be transferred between the GPU RAM and the registers in the streaming multiprocessors?  
> **username_2**: It's probably something like: the theoretical max rate at which data can be transferred from the GPU RAM into the GPU's L2 cache (or vice versa). This would set an upper limit on how quickly a rendering pass can run that needs to read a large amount of texture data (more than can fit into L2).  BTW, the bandwidth between GPU RAM and system RAM (for discrete GPU setups) is limited by the PCIe link bandwidth, which for PCIe Gen4 x16 is ~32 GB/s theoretical max (in practice somewhat less).  

---

## 2 answers

### Answer by username_3 on 2022-05-26. Score: 4

Before we begin, it's important to understand that raw spec numbers are typically not meant for use by actual graphics programmers. They are marketing tools to sell products. Sure, they have to be "accurate" (somewhat), but that does not mean that they are meaningful.

As for what bandwidth actually means, consider two parking lots. They can hold a number of cars. Connecting them is a road. This road has a number of lanes. In terms of meaningful "bandwidth," that is the answer to the question, "how many cars can I get from one lot to the other?" In our analogy, parking lots are pools of memory; the road is the connection between them.

So if someone tells you they have a 16 lane highway, does that mean something? Not really, not unless you know what parking lots that highway *connects*. The same goes for bandwidth; a raw number means nothing without knowing the source and destination memory for that bandwidth.

Marketers sometimes just add bandwidths from completely different memory pools together, as though the sum of lanes from 4 unconnected highways has some actual meaning.

So bandwidth is a measure of the rate of flow between pools of memory. It is the amount of data that can flow from one pool to another in a unit of time. These pools can be general GPU RAM, storage within a compute unit, various caches, etc. Many modern GPUs have memory architectures that connect many of these pools onto a single bus (effectively linking all parking lots by a single set of roads).

The other thing to remember is that pools of memory are not always *permanent*. Caches, compute storage, and the like are volatile; they get overwritten constantly by design. They exist because accessing bulk storage is slow, so operations that need to access memory quickly need small pools of faster memory closer to their use of this data.

If the bandwidth from GPU memory to a texture cache is 1'555GB/sec, this means that, within a 60fps frame, the total amount of storage that all shaders can access via texture fetches is 25.9GB. You may note that this is much smaller than the 40GB of storage such a GPU may have.

GPU bandwidths are so large *per second* because GPUs have to use that bandwidth many times per second. Once you look at them per frame, they're not so large.

Also, all bandwidth from one pool to another is purely *theoretical*. If nobody is reading from memory at any particular cycle, that cycle's worth of bandwidth goes *unused*. It can't be stored and use later when there's more memory pressure. You either use it or you lose it. And since GPU workloads tend to read memory in spurts, the *usable* bandwidth is often lower than the numerically available bandwidth.

> **Comments:**
> **username_0**: So do you agree with username_2 username_2 that "memory bandwidth" is the bandwidth between gpu memory and gpu L2 cache?  
> **username_3**: @username_0: No. "Memory bandwidth" as a term only makes sense in relation to the connection between two pools of memory. GPUs have *lots* of pools of memory. So the term "memory bandwidth" with no additional context is at best underspecified and should only be thought of as a general marketing buzzword, not a useful construct. They may have added together many actual bandwidth numbers to compute it. You don't know.  
> **username_3**: @username_0: Also, GPUs have *many* caches.  

---

### Answer by username_4 on 2022-05-26. Score: 0

Memory Bandwidth is the theoretical maximum amount of data that the bus can handle at any given time, playing a determining role in how quickly a GPU can access and utilize its frame buffer.


---

