<p>I am trying to perform quad-based tessellation using GLSL on Unity to be able to render polynomial-based parametric surfaces on a Unity quad. The quad-based tessellation using GLSL seems to be incorrectly choosing the vertices for the patch to be tessellated. I am using this shader (<a href="https://github.com/przemyslawzaworski/Unity3D-CG-programming/blob/master/GLSL/tessellation.shader" rel="nofollow noreferrer">https://github.com/przemyslawzaworski/Unity3D-CG-programming/blob/master/GLSL/tessellation.shader</a>) as a reference. However, modifying it to use quads in the domain shader stage messes up the ordering. </p>

<pre><code>Shader "Custom/Tessellation"
{
    Properties
    {
        _MainTex ("Texture", 2D) = "white" {}
        _factor("Tessellation scale",Range(1.0,64.0)) = 1.0
    }
    SubShader
    {
        Cull Off
        Pass
        {
            GLSLPROGRAM
            #version 460     
            uniform float _factor;

            #ifdef VERTEX
                in  vec4 in_POSITION0;
                void main()
                {
                    gl_Position =  gl_ModelViewProjectionMatrix * in_POSITION0;
                }
            #endif

            #ifdef HULL          //GLSL Tessellation Control Shader

                layout (vertices = 4) out;
                void main()
                {
                    if (gl_InvocationID == 0)
                    {
                        gl_TessLevelInner[0] = _factor;   //Inside tessellation factor
                        gl_TessLevelInner[1] = _factor;   //Inside tessellation factor

                        gl_TessLevelOuter[0] = _factor;   //Edge tessellation factor
                        gl_TessLevelOuter[1] = _factor;   //Edge tessellation factor
                        gl_TessLevelOuter[2] = _factor;   //Edge tessellation factor
                        gl_TessLevelOuter[3] = _factor;   //Edge tessellation factor
                    } 
                    gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;
                }
            #endif

            #ifdef DOMAIN        //GLSL Tessellation Evaluation Shader
                layout (quads) in;
                void main()
                {   
                    vec4 p1 = mix(gl_in[0].gl_Position, gl_in[1].gl_Position, gl_TessCoord.x);
                    vec4 p2 = mix(gl_in[2].gl_Position, gl_in[3].gl_Position, gl_TessCoord.x);
                    gl_Position = mix(p1, p2, gl_TessCoord.y);      
                }
            #endif

            #ifdef GEOMETRY      //geometry shader for rendering wireframe
                layout(triangles) in;
                layout(line_strip, max_vertices = 3) out;
                void main()
                {
                    for(int i = 0; i &lt; gl_in.length(); ++i)
                    {
                        gl_Position = gl_in[i].gl_Position;
                        EmitVertex();
                    }
                    gl_Position = gl_in[0].gl_Position;
                    EmitVertex();  
                    EndPrimitive();
                }    
            #endif

            #ifdef FRAGMENT
                out vec4 color;
                void main()
                {
                    color = vec4(1,1,1,1);
                }
            #endif

            ENDGLSL
        }
    }
}
</code></pre>

<p>Please help me in identifying what modifications are incorrect. I would appreciate any and all the help. The following are the outputs both from the triangle-based shader as given in the link to the Github page above, and the quad-based shader with modifications for quads made by me. </p>

<p><a href="https://i.sstatic.net/m8Cnk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m8Cnk.png" alt="Triangle-based Tessellation Working Properly"></a></p>

<p><a href="https://i.sstatic.net/KUUKV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KUUKV.png" alt="Quad-based Tessellation Generating Incorrect Results"></a></p>

CC BY-SA 4.0
2020-02-07T09:26:18.300Z
I've not looked at the code, but visually it looks to me that you're supplying it with, say, vertices in the order "ABCD" when it perhaps expects "ABDC" (or vice versa).
Simon F
CC BY-SA 4.0
2020-02-08T04:30:02.600Z
Oh yes @SimonF it does seem to get fixed on reordering the vertices in domain shader. However, an ordering only works with one model type on Unity for me. For instance if ABCD works for Planes, then maybe a different ordering like ADBC works for Cubes. Is there any way to have a consistent ordering across objects?
Bhavyansh Mishra
CC BY-SA 4.0
2020-02-11T09:12:07.113Z
I don't know unity, but I'd be surprised that vertex ordering would change between a flat quad and a cube... but perhaps there is an inconsistency in the system. I know of some Unity people on twitter, but I don't know if they also have a presence here.
Simon F
<shader><glsl><unity>
Problem with Quad-based Tessellation in GLSL generating incorrect vertex ordered tessellation
Bhavyansh Mishra
# Problem with Quad-based Tessellation in GLSL generating incorrect vertex ordered tessellation

(Asked by: username_0 on 2020-02-07)

I am trying to perform quad-based tessellation using GLSL on Unity to be able to render polynomial-based parametric surfaces on a Unity quad. The quad-based tessellation using GLSL seems to be incorrectly choosing the vertices for the patch to be tessellated. I am using this shader (<https://github.com/przemyslawzaworski/Unity3D-CG-programming/blob/master/GLSL/tessellation.shader>) as a reference. However, modifying it to use quads in the domain shader stage messes up the ordering.

```
Shader "Custom/Tessellation"
{
    Properties
    {
        _MainTex ("Texture", 2D) = "white" {}
        _factor("Tessellation scale",Range(1.0,64.0)) = 1.0
    }
    SubShader
    {
        Cull Off
        Pass
        {
            GLSLPROGRAM
            #version 460     
            uniform float _factor;

            #ifdef VERTEX
                in  vec4 in_POSITION0;
                void main()
                {
                    gl_Position =  gl_ModelViewProjectionMatrix * in_POSITION0;
                }
            #endif

            #ifdef HULL          //GLSL Tessellation Control Shader

                layout (vertices = 4) out;
                void main()
                {
                    if (gl_InvocationID == 0)
                    {
                        gl_TessLevelInner[0] = _factor;   //Inside tessellation factor
                        gl_TessLevelInner[1] = _factor;   //Inside tessellation factor

                        gl_TessLevelOuter[0] = _factor;   //Edge tessellation factor
                        gl_TessLevelOuter[1] = _factor;   //Edge tessellation factor
                        gl_TessLevelOuter[2] = _factor;   //Edge tessellation factor
                        gl_TessLevelOuter[3] = _factor;   //Edge tessellation factor
                    } 
                    gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;
                }
            #endif

            #ifdef DOMAIN        //GLSL Tessellation Evaluation Shader
                layout (quads) in;
                void main()
                {   
                    vec4 p1 = mix(gl_in[0].gl_Position, gl_in[1].gl_Position, gl_TessCoord.x);
                    vec4 p2 = mix(gl_in[2].gl_Position, gl_in[3].gl_Position, gl_TessCoord.x);
                    gl_Position = mix(p1, p2, gl_TessCoord.y);      
                }
            #endif

            #ifdef GEOMETRY      //geometry shader for rendering wireframe
                layout(triangles) in;
                layout(line_strip, max_vertices = 3) out;
                void main()
                {
                    for(int i = 0; i < gl_in.length(); ++i)
                    {
                        gl_Position = gl_in[i].gl_Position;
                        EmitVertex();
                    }
                    gl_Position = gl_in[0].gl_Position;
                    EmitVertex();  
                    EndPrimitive();
                }    
            #endif

            #ifdef FRAGMENT
                out vec4 color;
                void main()
                {
                    color = vec4(1,1,1,1);
                }
            #endif

            ENDGLSL
        }
    }
}

```

Please help me in identifying what modifications are incorrect. I would appreciate any and all the help. The following are the outputs both from the triangle-based shader as given in the link to the Github page above, and the quad-based shader with modifications for quads made by me.

[![Triangle-based Tessellation Working Properly](https://i.sstatic.net/m8Cnk.png)](https://i.sstatic.net/m8Cnk.png)

[![Quad-based Tessellation Generating Incorrect Results](https://i.sstatic.net/KUUKV.png)](https://i.sstatic.net/KUUKV.png)

**Tags:** shader, glsl, unity

> **Comments:**
> **username_1**: I've not looked at the code, but visually it looks to me that you're supplying it with, say, vertices in the order "ABCD" when it perhaps expects "ABDC" (or vice versa).  
> **username_0**: Oh yes @username_1 it does seem to get fixed on reordering the vertices in domain shader. However, an ordering only works with one model type on Unity for me. For instance if ABCD works for Planes, then maybe a different ordering like ADBC works for Cubes. Is there any way to have a consistent ordering across objects?  
> **username_1**: I don't know unity, but I'd be surprised that vertex ordering would change between a flat quad and a cube... but perhaps there is an inconsistency in the system. I know of some Unity people on twitter, but I don't know if they also have a presence here.  

---

## 0 answers

