<p>For Nvidia GPUs there are several machine languages all of which can be read here:
<a href="https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html#instruction-set-ref" rel="noreferrer">https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html#instruction-set-ref</a></p>
<p>Above the &quot;Instruction Set Reference&quot; section there are also some examples on SM70 variant of SASS code which is stored in ELF and dumped from there too so it's a machine code as ELF always was an unmanaged code container by historic reasons (apart from PE which could contain CIL managed code but even in this case it must be in data section while ELF does not support such confusing feature at all) and it's not managed/virtual bytecode or something as some users say so, it's a real native compiled device code for GPU. The correct instructions though are those which are written with capital letter and addresses in hexademical form (like /*0000*/ MOV R1, c[0x0][0x28] ;), others are just PTX and they aren't actually relevant to the question. As a drawback, fatbin technique may be required if a distributor has chosen to publish the GPU program in precompiled form rather than source code or at least portable PTX. But that is very rare as there are not as much such programs around as binaries for CPU. Anyway such code is usually contained within cubin files which are dynamically loaded by calling &quot;cuModuleLoad&quot; then you rewrite pointer to device GPU function handle using &quot;cuModuleGetFunction&quot; and then call &quot;cuLaunchKernel&quot; which forces the driver to send binary code to the GPU and start execution from like 0 address as all the kernels start from 0 (actually ELF variant used by Nvidia contains many sections but the driver only sends the desired kernel code along with its subroutines and data such as switch data etc.). And what about the instructions itself, well, they are mostly RISC-fashioned. There is no div or mod, instead nvcc compiler must output sophisticated algorithm based on addition, subtraction, multiplication, shifts, bitwise logical operations (bitwise and mostly). Only addressing mode reminds some CISC presence like &quot;LD R2, [R2+0x1c];&quot; etc.</p>

CC BY-SA 4.0
2020-09-29T08:02:32.197Z
Your answer would easier to understand if you a) used far fewer conjunctions in each sentence and b) Put in some paragraphs.
Simon F
CC BY-SA 4.0
2020-09-16T20:57:30.947Z
2020-09-16T21:06:30.297Z
2020-09-16T21:06:30.297Z
Danil
<p>What you've read is correct. In general, GPU vendors do not release lists of machine instructions for their GPUs. </p>

<p>That said, you can do something similar to assembly programming on the GPU by using <a href="https://en.wikipedia.org/wiki/ARB_assembly_language" rel="noreferrer">OpenGL ARB Assembly Language</a>. It allows you to program in assembly style, writing 1 opcode and its operands per line. While this is not quite the same as writing assembly language for the CPU, it's as close as your likely to get. I believe it's deprecated in modern OpenGL (though not positive). It still worked as of OpenGL 2.1.</p>

CC BY-SA 4.0
2018-07-16T21:28:55.600Z
I'll take a look. Thanks for the infos and the link :).
sebastien finor
CC BY-SA 4.0
2018-07-15T05:31:39.157Z
2018-07-15T05:31:39.157Z
user1118321
<p>As others have said, GPUs expose a higher-level language, which allows multiple different architectures i.e. different vendors and different GPU generations, to all support the same applications.</p>
<p><em>[Update: Link to older ISA replaced with newer one]</em><br>
If, however, you're still curious, documentation for the recent <a href="https://docs.imgtec.com/Architecture_Guides/PowerVR_ISR/topics/d0e75.html" rel="nofollow noreferrer">PowerVR instruction set is available</a> - my favourite opcodes being &quot;FRED&quot; and &quot;FARCTANC&quot; :-)</p>
<p>Note that instructions sets can change from generation to generation so this is mainly for illustrative purposes.</p>

CC BY-SA 4.0
2018-07-16T09:46:40.387Z
2020-09-23T08:54:53.390Z
2020-09-23T08:54:53.390Z
Simon F
<p>You're tilting at windmills trying to learn "GPU assembly", and it's due to the differences between how CPUs and GPUs are made and sold.</p>

<p>Each CPU has what's called an <em>instruction set architecture</em>, for example x86 or ARMv8. The instruction set is the interface between the user of the CPU (i.e. the programmer) and the chip. The chip designer publishes the details of the instruction set so that compiler vendors can write compilers to target that instruction set. Any CPU that uses that instruction set can run the same binaries. (Extensions like SSE make that slightly untrue, but they only add new instructions: they don't change the structure of the binary.) When the vendor creates a new processor family, it could have completely different <em>micro-architecture</em> internally, but the same instruction set.</p>

<p>GPUs are not like this at all. For best efficiency, the instruction set is usually closely tied to the micro-architecture of the CPU. That means each new family of GPUs has a new instruction set. GPUs can't run binaries made for different GPUs. For this reason, the instruction set usually isn't published: instead, your interface to the GPU is the driver published by the vendor for each graphics API (OpenGL, Vulkan, DirectX, &amp;c.). This is why the graphics APIs have functions to take the source code of a shader and run it: the compiled shader only runs on the same model or family of GPU it was compiled for.</p>

<p>The closest you get to GPU assembly language is <em>SPIR-V</em>. This is an industry-standard intermediate representation, which GPU vendors are starting to support. It's like a GPU equivalent of LLVM's intermediate representation. It allows you to do the parsing and source optimization parts of compilation up-front to get a SPIR-V file, and then the GPU driver only needs to compile that to the GPU's instruction set at load time.</p>

CC BY-SA 4.0
2018-07-16T21:25:05.737Z
Thanks for your answer! It was rally helpful to me :). About the part with the shader example, I'm not sure if I understood well, but I think you meant that the shader, instead of being compiled and assembled, is only compiled and not assembled because instruction sets are different for each GPU, even from the same "family".
sebastien finor
CC BY-SA 4.0
2018-07-17T08:17:21.873Z
@sebastienfinor What I'm trying to say about that is that there isn't necessarily an intermediate "assembly language" representation in the compilation process.
Dan Hulme
CC BY-SA 4.0
2020-07-23T00:19:52.230Z
When programmable shaders were first being introduced they were programmed using a language that was very "assembly language" like. It can still be found in books and papers of the time. Some of the game programming gems books have articles that show examples. The original "orange book" has some in it as well. It is fun to look at because you will see early versions of functions like fma. It also can fill in some of those "why the heck is this done this way" questions. AND, you will quickly see that different vendors had different "languages".
pmw1234
CC BY-SA 4.0
2020-07-23T00:32:07.100Z
There is in fact still multiple layers, but programmers don't really have access to those layers. For example GLSL compiles to SPIR-V (as already mentioned) And those are really the only layers of the language you have access to as a graphics programmer. But inside the driver code you will find  mountains of vendor specific code...when this code actually gets inside the gpu it is further translated into micro code, micro code is where it all really happens, beyond that it is just a sea of numbers. Micro code is how vendors "fix" hardware bugs. By programming around them.
pmw1234
CC BY-SA 4.0
2021-04-29T16:17:11.060Z
"_What I'm trying to say about that is that there isn't necessarily an intermediate "assembly language" representation in the compilation process_" What does this mean precisely? It all eventually becomes zeroes & ones, right? Right? `;^D` Even if each card has its own instruction set (though who would reinvent `lda`, `ina`, `sta`?), you should be able to access that instruction set somehow, right? Or do GPU shops write drivers and keep the metal proprietary? (Though I still bet someone's tried to reverse engineer a few cards.) Assembly is just labels for bytes. Don't GPUs use bytes? /confused
ruffin
CC BY-SA 4.0
2018-07-15T08:27:37.640Z
2018-07-15T08:27:37.640Z
Dan Hulme
<p>I have played around with CPU assembly programming like Nasm, Tasm or Masm, but I'm really curious to know how GPU works now.
However, i'm quite confused when I look on internet. I've heard about Cuda and OpenCL, but it is not what I'm looking for.
I'd like to know how GPUs instructions are in RAM... What are the Nasm and Masm for most GPUs? What is the x86 or Z80 of GPUs (What are the different families of GPU)? Do you know a constructor opcodes reference manual?
I think I really need something to compare between the two Processing Units to make it clear, because GPU assembly programming seems to be an even harder subject to learn from on internet that CPU asm programming. I've also read that &quot;NVIDIA never released details on the instructions actually understood by their hardware&quot;, but it seems pretty surprising to me. Full post there: <a href="https://stackoverflow.com/questions/4660974/how-to-create-or-manipulate-gpu-assembler?newreg=e31519279ce949f087df6322dbf2bf4d">https://stackoverflow.com/questions/4660974/how-to-create-or-manipulate-gpu-assembler?newreg=e31519279ce949f087df6322dbf2bf4d</a></p>
<p>Thanks for your help!</p>

CC BY-SA 4.0
2018-07-15T20:32:14.387Z
[This blog post](https://anteru.net/blog/2018/more-compute-shaders/) offers a peek at what AMD GCN assembly looks like (which, unlike some other GPU vendors, is actually documented in public).
Nathan Reed
CC BY-SA 4.0
2019-10-02T13:16:49.660Z
Also https://stackoverflow.com/questions/1697842/do-graphic-cards-have-instruction-sets-of-their-own and https://stackoverflow.com/questions/7353136/is-there-an-assembly-language-for-cuda
Ciro Santilli OurBigBook.com
<gpu><assembly>
What does GPU assembly look like?
sebastien finor
# What does GPU assembly look like?

(Asked by: username_0 on 2018-07-14)

I have played around with CPU assembly programming like Nasm, Tasm or Masm, but I'm really curious to know how GPU works now.
However, i'm quite confused when I look on internet. I've heard about Cuda and OpenCL, but it is not what I'm looking for.
I'd like to know how GPUs instructions are in RAM... What are the Nasm and Masm for most GPUs? What is the x86 or Z80 of GPUs (What are the different families of GPU)? Do you know a constructor opcodes reference manual?
I think I really need something to compare between the two Processing Units to make it clear, because GPU assembly programming seems to be an even harder subject to learn from on internet that CPU asm programming. I've also read that "NVIDIA never released details on the instructions actually understood by their hardware", but it seems pretty surprising to me. Full post there: <https://stackoverflow.com/questions/4660974/how-to-create-or-manipulate-gpu-assembler?newreg=e31519279ce949f087df6322dbf2bf4d>

Thanks for your help!

**Tags:** gpu, assembly

> **Comments:**
> **username_1**: [This blog post](https://anteru.net/blog/2018/more-compute-shaders/) offers a peek at what AMD GCN assembly looks like (which, unlike some other GPU vendors, is actually documented in public).  
> **username_2**: Also https://stackoverflow.com/questions/1697842/do-graphic-cards-have-instruction-sets-of-their-own and https://stackoverflow.com/questions/7353136/is-there-an-assembly-language-for-cuda  

---

## 4 answers

### Answer by username_6 on 2018-07-15. Score: 33 | Accepted answer

You're tilting at windmills trying to learn "GPU assembly", and it's due to the differences between how CPUs and GPUs are made and sold.

Each CPU has what's called an *instruction set architecture*, for example x86 or ARMv8. The instruction set is the interface between the user of the CPU (i.e. the programmer) and the chip. The chip designer publishes the details of the instruction set so that compiler vendors can write compilers to target that instruction set. Any CPU that uses that instruction set can run the same binaries. (Extensions like SSE make that slightly untrue, but they only add new instructions: they don't change the structure of the binary.) When the vendor creates a new processor family, it could have completely different *micro-architecture* internally, but the same instruction set.

GPUs are not like this at all. For best efficiency, the instruction set is usually closely tied to the micro-architecture of the CPU. That means each new family of GPUs has a new instruction set. GPUs can't run binaries made for different GPUs. For this reason, the instruction set usually isn't published: instead, your interface to the GPU is the driver published by the vendor for each graphics API (OpenGL, Vulkan, DirectX, &c.). This is why the graphics APIs have functions to take the source code of a shader and run it: the compiled shader only runs on the same model or family of GPU it was compiled for.

The closest you get to GPU assembly language is *SPIR-V*. This is an industry-standard intermediate representation, which GPU vendors are starting to support. It's like a GPU equivalent of LLVM's intermediate representation. It allows you to do the parsing and source optimization parts of compilation up-front to get a SPIR-V file, and then the GPU driver only needs to compile that to the GPU's instruction set at load time.

> **Comments:**
> **username_0**: Thanks for your answer! It was rally helpful to me :). About the part with the shader example, I'm not sure if I understood well, but I think you meant that the shader, instead of being compiled and assembled, is only compiled and not assembled because instruction sets are different for each GPU, even from the same "family".  
> **username_6**: @username_0 What I'm trying to say about that is that there isn't necessarily an intermediate "assembly language" representation in the compilation process.  
> **username_7**: When programmable shaders were first being introduced they were programmed using a language that was very "assembly language" like. It can still be found in books and papers of the time. Some of the game programming gems books have articles that show examples. The original "orange book" has some in it as well. It is fun to look at because you will see early versions of functions like fma. It also can fill in some of those "why the heck is this done this way" questions. AND, you will quickly see that different vendors had different "languages".  
> **username_7**: There is in fact still multiple layers, but programmers don't really have access to those layers. For example GLSL compiles to SPIR-V (as already mentioned) And those are really the only layers of the language you have access to as a graphics programmer. But inside the driver code you will find  mountains of vendor specific code...when this code actually gets inside the gpu it is further translated into micro code, micro code is where it all really happens, beyond that it is just a sea of numbers. Micro code is how vendors "fix" hardware bugs. By programming around them.  
> **username_8**: "_What I'm trying to say about that is that there isn't necessarily an intermediate "assembly language" representation in the compilation process_" What does this mean precisely? It all eventually becomes zeroes & ones, right? Right? `;^D` Even if each card has its own instruction set (though who would reinvent `lda`, `ina`, `sta`?), you should be able to access that instruction set somehow, right? Or do GPU shops write drivers and keep the metal proprietary? (Though I still bet someone's tried to reverse engineer a few cards.) Assembly is just labels for bytes. Don't GPUs use bytes? /confused  

---

### Answer by username_3 on 2020-09-16. Score: 5

For Nvidia GPUs there are several machine languages all of which can be read here:
<https://docs.nvidia.com/cuda/cuda-binary-utilities/index.html#instruction-set-ref>

Above the "Instruction Set Reference" section there are also some examples on SM70 variant of SASS code which is stored in ELF and dumped from there too so it's a machine code as ELF always was an unmanaged code container by historic reasons (apart from PE which could contain CIL managed code but even in this case it must be in data section while ELF does not support such confusing feature at all) and it's not managed/virtual bytecode or something as some users say so, it's a real native compiled device code for GPU. The correct instructions though are those which are written with capital letter and addresses in hexademical form (like /\*0000\*/ MOV R1, c[0x0][0x28] ;), others are just PTX and they aren't actually relevant to the question. As a drawback, fatbin technique may be required if a distributor has chosen to publish the GPU program in precompiled form rather than source code or at least portable PTX. But that is very rare as there are not as much such programs around as binaries for CPU. Anyway such code is usually contained within cubin files which are dynamically loaded by calling "cuModuleLoad" then you rewrite pointer to device GPU function handle using "cuModuleGetFunction" and then call "cuLaunchKernel" which forces the driver to send binary code to the GPU and start execution from like 0 address as all the kernels start from 0 (actually ELF variant used by Nvidia contains many sections but the driver only sends the desired kernel code along with its subroutines and data such as switch data etc.). And what about the instructions itself, well, they are mostly RISC-fashioned. There is no div or mod, instead nvcc compiler must output sophisticated algorithm based on addition, subtraction, multiplication, shifts, bitwise logical operations (bitwise and mostly). Only addressing mode reminds some CISC presence like "LD R2, [R2+0x1c];" etc.

> **Comments:**
> **username_4**: Your answer would easier to understand if you a) used far fewer conjunctions in each sentence and b) Put in some paragraphs.  

---

### Answer by username_5 on 2018-07-15. Score: 5

What you've read is correct. In general, GPU vendors do not release lists of machine instructions for their GPUs.

That said, you can do something similar to assembly programming on the GPU by using [OpenGL ARB Assembly Language](https://en.wikipedia.org/wiki/ARB_assembly_language). It allows you to program in assembly style, writing 1 opcode and its operands per line. While this is not quite the same as writing assembly language for the CPU, it's as close as your likely to get. I believe it's deprecated in modern OpenGL (though not positive). It still worked as of OpenGL 2.1.

> **Comments:**
> **username_0**: I'll take a look. Thanks for the infos and the link :).  

---

### Answer by username_4 on 2018-07-16. Score: 4

As others have said, GPUs expose a higher-level language, which allows multiple different architectures i.e. different vendors and different GPU generations, to all support the same applications.

*[Update: Link to older ISA replaced with newer one]*  
If, however, you're still curious, documentation for the recent [PowerVR instruction set is available](https://docs.imgtec.com/Architecture_Guides/PowerVR_ISR/topics/d0e75.html) - my favourite opcodes being "FRED" and "FARCTANC" :-)

Note that instructions sets can change from generation to generation so this is mainly for illustrative purposes.


---

