<blockquote>
<p>Now, when reading other questions and articles about it, they always say, any simple operation, + - * / , would be a floating-point operation, which makes sense in terms of the word itself</p>
</blockquote>
<p>This is basically what is meant when talking about floating-point operations although I would exclude the division since it takes much longer than the other operations (see below).</p>
<blockquote>
<p>BUT: A multiplication is (obviously) often &quot;far&quot; more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)</p>
</blockquote>
<p>Most modern processors can usually calculate additions and multiplications equally fast. Even though the numbers are for CPUs and not for GPUs, you can take a look at the corresponding commands in the <a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/" rel="nofollow noreferrer">Intel intrinsics guide</a>. A GPU shouldn't behave too different in this matter. Here are some examples:</p>
<ul>
<li><a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_add_ps&amp;expand=3928,3928,133" rel="nofollow noreferrer">addition</a></li>
<li><a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_sub_ps&amp;expand=3928,3928,133,5727" rel="nofollow noreferrer">subtraction</a></li>
<li><a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_mul_ps&amp;expand=3928,3928" rel="nofollow noreferrer">multiplication</a></li>
<li><a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_div_ps&amp;expand=3928,3928,133,5727,2156" rel="nofollow noreferrer">division</a></li>
</ul>
<p>As you can see, the numbers of the Skylake processor are all identical except for the division.</p>
<p>In case you wonder what those numbers mean: Latency is the number of cycles that are necessary to calculate the result and throughput is the inverse number of operations you can start per cycle. So for addition, subtraction and multiplication, a skylake processor can start the calculation of 2 operations per cycle and get the results 4 cycles later. This means that you can actually have 8 of those operations in progress if they do not depend on each other.</p>
<p>Knowing this it is easy to see, that divisions take much longer to compute. This is why you should always calculate the inverse and multiply it instead if you need to divide a bunch of numbers by the same value.</p>
<blockquote>
<p>BUT: A multiplication is (obviously) often &quot;far&quot; more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)</p>
<p>Following, just a simple example for demonstration:</p>
<p>5+5= 5+5</p>
<p>5*5= 5+5+5+5+5</p>
</blockquote>
<p>Well, this would be very inefficient. If you are curious how a computer actually performs a multiplication efficiently, take a look into <a href="https://stackoverflow.com/questions/3060064/how-computer-multiplies-2-numbers">this question</a></p>

CC BY-SA 4.0
2021-01-29T12:18:11.417Z
Thank you for the in-depth answer.
I though that a PC would maybe calculate multiplications differently, but wasn't sure, that's also good to know.
A very helpful answer:)
LuckyLuke Skywalker
CC BY-SA 4.0
2021-01-29T14:00:51.477Z
*" take a look into this question"*  Ahhh, but that leaves out the joys of "modified booth encoding" and "carry save adders" etc :-)
Simon F
CC BY-SA 4.0
2021-01-29T15:11:23.810Z
And if you want to be impressed take a look at this monster:https://www.hpe.com/us/en/newsroom/press-release/2020/03/hpe-and-amd-power-complex-scientific-discovery-in-worlds-fastest-supercomputer-for-us-department-of-energys-doe-national-nuclear-security-administration-nnsa.html
pmw1234
CC BY-SA 4.0
2021-01-29T12:05:16.953Z
2021-01-29T12:05:16.953Z
wychmaster
<p>Just to add to <a href="https://computergraphics.stackexchange.com/a/10606/209">whychmaster's reply</a>, in my experience, on a GPU, a floating-point operation, when used in quoting FLOPS (floating-point operations per second) benchmark figures will refer to Addition (equiv subtraction) and Multiplication.</p>
<p>The other standard IEEE operators, division + sqrt, are more computationally expensive and so unlikely to be included in the figure. <em>(Note that, there is often HW support in GPUs for accelerating operations like 1/x, 1/sqrt(x) etc, perhaps to make them as (or nearly as) fast as muls &amp; adds)</em></p>
<p>Having said that, it is very likely that a GPU will actually implement a Fused Multipy-Add operation, FMAD, i.e. do  A * B + C in &quot;one step&quot; as this is</p>
<ol>
<li>more accurate than doing separate multiply and add instructions</li>
<li>is faster than separate ops</li>
<li>is about the same silicon cost</li>
<li>is a very common building block of CG operations (e.g dot products, convolutions)</li>
</ol>
<p>I suspect that in quoted figures, though, an FMAD would be considered 2 operations.</p>

CC BY-SA 4.0
2021-01-29T14:18:11.680Z
2021-01-29T14:18:11.680Z
Simon F
<p>I spent many years doing hardware work creating CPU's for the &quot;big iron&quot; market. We considered a FLOP to be any instruction that requires the &quot;floating point execution unit&quot; inside the CPU to perform the calculation that another instruction has a dependency on. This unit used to be considered one of the most complex, slowest, and one of the most power hungry units inside the CPU. It could result in long stalls, lots of NOPS being inserted into the pipeline, lots of heat being generated, and the electric bill to go sky high.</p>
<p>Much research later, the floating point units are much less power hungry, and can often be compared to the performance of a good integer execution unit. (last I heard research was combining the two into a flexible integer/floating unit)</p>
<p>This all lead to measuring how many FLOPS a CPU (and nowadays the GPU) can perform in any given second, and more generally an entire system with many CPU's and GPU's working together. This measurement is still relevant because GPU's have turned number crunching into a fine art and while throughput is often dependent on memory architecture the raw number of FLOPs still tends to be a good measurement but can be very workflow dependent.</p>

CC BY-SA 4.0
2021-02-03T15:30:15.483Z
2021-02-03T15:30:15.483Z
pmw1234
<p>I am interested in how much a GPU can actually calculate. When learning about rendering, it seems like a good idea as well.</p>
<p>My Question:
<strong>WHAT EXACTLY is a Floating Point Operation?</strong></p>
<p>I know what a floating point number is.</p>
<p>Now, when reading other questions and articles about it, they always say, any simple operation, + - * / , would be a floating point operation, which makes sense in terms of the word itself</p>
<p>BUT:   <em>A multiplication is (obviously) often &quot;far&quot; more (calculating) than an addition, isn't it?!</em>
(Apart from the sheer size of the calculation)</p>
<p>Following, just a simple example for demonstration:</p>
<p>5+5=       5+5</p>
<p>5*5=       5+5+5+5+5</p>
<p>So does the term &quot;flop&quot; actually just describe additions and subtractions
or is a flop in the end just a very rough definition?
Or is the GPU somehow equally fast at all of those calculations?</p>
<p>Thanks in advance.</p>

<gpu><hardware><theory>
FLOP - exact definition
LuckyLuke Skywalker
# FLOP - exact definition

(Asked by: username_0 on 2021-01-29)

I am interested in how much a GPU can actually calculate. When learning about rendering, it seems like a good idea as well.

My Question:
**WHAT EXACTLY is a Floating Point Operation?**

I know what a floating point number is.

Now, when reading other questions and articles about it, they always say, any simple operation, + - \* / , would be a floating point operation, which makes sense in terms of the word itself

BUT: *A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?!*
(Apart from the sheer size of the calculation)

Following, just a simple example for demonstration:

5+5= 5+5

5\*5= 5+5+5+5+5

So does the term "flop" actually just describe additions and subtractions
or is a flop in the end just a very rough definition?
Or is the GPU somehow equally fast at all of those calculations?

Thanks in advance.

**Tags:** gpu, hardware, theory


---

## 3 answers

### Answer by username_1 on 2021-01-29. Score: 0 | Accepted answer

> Now, when reading other questions and articles about it, they always say, any simple operation, + - \* / , would be a floating-point operation, which makes sense in terms of the word itself

This is basically what is meant when talking about floating-point operations although I would exclude the division since it takes much longer than the other operations (see below).

> BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)

Most modern processors can usually calculate additions and multiplications equally fast. Even though the numbers are for CPUs and not for GPUs, you can take a look at the corresponding commands in the [Intel intrinsics guide](https://software.intel.com/sites/landingpage/IntrinsicsGuide/). A GPU shouldn't behave too different in this matter. Here are some examples:

* [addition](https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_add_ps&expand=3928,3928,133)
* [subtraction](https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_sub_ps&expand=3928,3928,133,5727)
* [multiplication](https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_mul_ps&expand=3928,3928)
* [division](https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_div_ps&expand=3928,3928,133,5727,2156)

As you can see, the numbers of the Skylake processor are all identical except for the division.

In case you wonder what those numbers mean: Latency is the number of cycles that are necessary to calculate the result and throughput is the inverse number of operations you can start per cycle. So for addition, subtraction and multiplication, a skylake processor can start the calculation of 2 operations per cycle and get the results 4 cycles later. This means that you can actually have 8 of those operations in progress if they do not depend on each other.

Knowing this it is easy to see, that divisions take much longer to compute. This is why you should always calculate the inverse and multiply it instead if you need to divide a bunch of numbers by the same value.

> BUT: A multiplication is (obviously) often "far" more (calculating) than an addition, isn't it?! (Apart from the sheer size of the calculation)
>
> Following, just a simple example for demonstration:
>
> 5+5= 5+5
>
> 5\*5= 5+5+5+5+5

Well, this would be very inefficient. If you are curious how a computer actually performs a multiplication efficiently, take a look into [this question](https://stackoverflow.com/questions/3060064/how-computer-multiplies-2-numbers)

> **Comments:**
> **username_0**: Thank you for the in-depth answer. I though that a PC would maybe calculate multiplications differently, but wasn't sure, that's also good to know. A very helpful answer:)  
> **username_2**: *" take a look into this question"*  Ahhh, but that leaves out the joys of "modified booth encoding" and "carry save adders" etc :-)  
> **username_3**: And if you want to be impressed take a look at this monster:https://www.hpe.com/us/en/newsroom/press-release/2020/03/hpe-and-amd-power-complex-scientific-discovery-in-worlds-fastest-supercomputer-for-us-department-of-energys-doe-national-nuclear-security-administration-nnsa.html  

---

### Answer by username_2 on 2021-01-29. Score: 1

Just to add to [whychmaster's reply](https://computergraphics.stackexchange.com/a/10606/209), in my experience, on a GPU, a floating-point operation, when used in quoting FLOPS (floating-point operations per second) benchmark figures will refer to Addition (equiv subtraction) and Multiplication.

The other standard IEEE operators, division + sqrt, are more computationally expensive and so unlikely to be included in the figure. *(Note that, there is often HW support in GPUs for accelerating operations like 1/x, 1/sqrt(x) etc, perhaps to make them as (or nearly as) fast as muls & adds)*

Having said that, it is very likely that a GPU will actually implement a Fused Multipy-Add operation, FMAD, i.e. do A \* B + C in "one step" as this is

1. more accurate than doing separate multiply and add instructions
2. is faster than separate ops
3. is about the same silicon cost
4. is a very common building block of CG operations (e.g dot products, convolutions)

I suspect that in quoted figures, though, an FMAD would be considered 2 operations.


---

### Answer by username_3 on 2021-02-03. Score: 0

I spent many years doing hardware work creating CPU's for the "big iron" market. We considered a FLOP to be any instruction that requires the "floating point execution unit" inside the CPU to perform the calculation that another instruction has a dependency on. This unit used to be considered one of the most complex, slowest, and one of the most power hungry units inside the CPU. It could result in long stalls, lots of NOPS being inserted into the pipeline, lots of heat being generated, and the electric bill to go sky high.

Much research later, the floating point units are much less power hungry, and can often be compared to the performance of a good integer execution unit. (last I heard research was combining the two into a flexible integer/floating unit)

This all lead to measuring how many FLOPS a CPU (and nowadays the GPU) can perform in any given second, and more generally an entire system with many CPU's and GPU's working together. This measurement is still relevant because GPU's have turned number crunching into a fine art and while throughput is often dependent on memory architecture the raw number of FLOPs still tends to be a good measurement but can be very workflow dependent.


---

